A method for detecting internal defects in cable conduits guided by saliency perception

Abstract

The present invention discloses a method for detecting internal defects of cable conduits guided by saliency perception. The method comprises the following steps: first, extracting a visible light edge image by using the EDTER edge detection algorithm, and optimizing a depth image according to edge saliency information to obtain a depth optimized image; second, registering the visible light edge image and the depth optimized image according to a coordinate transformation relationship; then, based on the weak edge features inside the cable conduit, using a pipeline internal defect detection network based on RGB-D information collaboration to extract depth saliency features and edge features, and detecting internal defects in the conduit and their locations; finally, based on the depth features of the defects, using a pipeline internal defect classification network guided by depth saliency information to classify the defect detection results. The present invention provides a method for detecting internal defects of cable conduits with high accuracy and good robustness, which can effectively improve the accuracy and efficiency of internal defect detection in cable conduits and has practical application value.

Description

Technical Field

[0001] The invention relates to a method for detecting internal defects of a cable conduit guided by significant perception, and belongs to the technical field of image processing. Background Art

[0002] With the rapid development of the national economy and urban construction, the demand for electricity is also increasing. Power grid construction in various cities is accelerating, and the number of cables is increasing year by year, leading to urban congestion. Due to the scarcity of land resources and the need for urban aesthetics, power cable lines are being used on a large scale. Compared to overhead lines, underground power cables do not occupy ground space and require dedicated cable tunnels or ducts. The use of underground power cable lines not only does not affect the aesthetics of the city, but also protects against the impact of natural disasters such as wind, rain, lightning, and pollution, effectively ensuring a safe and stable supply of electricity. To fully utilize land development value and improve urban beautification, overhead power transmission lines are gradually being replaced by underground cable lines within urban areas.

[0003] Conduit installation is a common method for laying cables. Underground conduits are pre-laid and cables are installed when needed. When installing the conduit, poor sealing at the joints can lead to concrete seeping into the conduits, forming sharp cones. Furthermore, unthreaded cable conduits can become damaged or infiltrated by foreign objects during idle time due to factors such as construction work or other natural factors. This can severely hinder cable conduit installation and pose a safety hazard. Underground cable conduits are widely distributed, and accidents can not only cause significant economic losses but can also cause electrical discharges, tripping circuits, and fires, posing a serious threat to the lives and property of surrounding residents.

[0004] With the rapid development of intelligent robotics and image processing technologies, robots equipped with image sensors can replace manual inspections, effectively improving the efficiency and safety of underground cable pipeline inspections. Based on the internal pipeline images captured by sensors, image processing techniques can be used to detect defects or foreign objects. Existing methods for detecting internal pipeline defects mostly rely on visible light image analysis and processing. However, the limited space, complex internal conditions, and poor lighting conditions of cable pipelines result in low visible light image quality. Defect detection methods based on visible light images have low accuracy and efficiency, and are unable to accurately analyze internal pipeline defects. Chinese patent publication number CN114266964A discloses a "Deep Image-Guided Cable Pipeline Defect Detection and Localization Method," which proposes a method for detecting and localizing internal defects in power pipelines by adaptively fusing visible light and depth image features. However, this method is slow in directly extracting and fusing visible light and depth image features and is susceptible to image noise, making it difficult to achieve accurate and efficient internal defect detection in inductive pipelines. To address this issue, this paper proposes a saliency-sensing-guided method for detecting internal defects in cable conduits. This method utilizes two different image modalities—visible light and depth images—to improve the accuracy and efficiency of defect detection by introducing edge information and optimizing depth image noise. This method enables precise and efficient detection of internal defects in cable conduits. This effectively prevents hidden cable conduit failures and improves the safety and stability of power system operations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to propose a significant perception-guided cable duct internal defect detection method, effectively improve the accuracy and efficiency of cable duct internal defect detection, realize accurate detection of cable duct internal defects, and correctly classify different types of defects.

[0006] To solve the above technical problems, the present invention provides a method for detecting internal defects of cable conduits guided by significant perception, comprising the following steps:

[0007] Step 1. Use a visible light camera and a depth camera based on structured light to obtain visible light images of the interior of the pipeline. RGB and depth image I depth ;

[0008] Step 2. Extract edge information of the visible light image using the EDTER edge detection algorithm to obtain a visible light edge image;

[0009] Step 3. Use the visible light edge image to optimize the original depth image to obtain a depth optimized image;

[0010] Step 4. Use a registration algorithm based on different coordinate system transformations to register the visible light edge image and the depth optimized image;

[0011] Step 5. Utilizing the complementary information of visible light and depth images, a pipeline internal defect detection network based on RGB-D information collaboration is used to detect internal defects in pipelines based on visible light edge images and depth-optimized images. The network then outputs a binary image of the detection results, achieving accurate detection of internal defects in pipelines.

[0012] Step 6. Based on the defect detection results, the pipeline internal defect classification network guided by deep saliency information is used to classify the defect detection results and correctly identify the internal defect category of the pipeline.

[0013] Furthermore, in step 2, edge information of the visible light image is extracted using the EDTER edge detection algorithm, and the specific process of obtaining the visible light edge image is as follows:

[0014] Step 2.1: Input visible light image I RGB Encode global context information: First, the input image is segmented into a series of coarse-grained image sequences and a global attention encoder is used to learn global context information; then a global bidirectional multi-stage aggregation decoder is used to generate a global feature map;

[0015] Step 2.2: Input visible light image I RGB Encoding local context information: First, the input image is sampled using a non-overlapping sliding window to divide the entire image into multiple fine-grained image block sequences; then, a local attention encoder is performed on each image block sequence to capture short-range local clues; finally, all features are input into a local bidirectional multi-stage aggregation decoder to generate a local feature map;

[0016] Step 2.3, global feature map f g and local feature f l Processed by the feature fusion module FFM and then input into the edge prediction network L h To generate the final edge map.

[0017] Furthermore, in step 2.1, the visible light image I RGB The specific process of encoding global context information is as follows:

[0018] Step 2.1.1. Visible light image I RGB The size is H×W, the number of channels is 3, and the image is represented by I RGB ∈R H ×W×3 ; First, evenly divide the image and input the two-dimensional image I RGB ∈R H×W×3Convert it into a one-dimensional image block sequence to obtain a coarse-grained image block sequence of size 16×16; then map the one-dimensional image block sequence to the potential embedding space through linear projection, and add the spatial position information corresponding to the original image in each image block to obtain the embedded variable

[0019] Step 2.1.2: Embed variables Input global attention encoder G E , calculate the global attention information, the calculation expression is:

[0020]

[0021] Among them, the encoder G E The output feature sequence is expressed as Lg is the encoder G E Output number, set Lg = 24, C is the number of feature channels;

[0022] Step 2.1.3, the feature sequence z g Input is a global attention decoder G with bidirectional multi-level aggregation D Upsampling is performed to obtain high-resolution features; decoder G D A bidirectional feature aggregation strategy is used, including a top-down path that fuses low-level features from high-level features and a bottom-up path that fuses high-level features from low-level features;

[0023] First, the feature sequence Divide into four groups and take the last feature of each feature sequence As decoder G D Input; after passing through the convolution layer with a convolution kernel size of 1×1, the feature map z g From the two-dimensional expression R Restore to three-dimensional expression R The number of channels remains unchanged;

[0024] Then, aggregate features from top to bottom: After inputting into the convolution layer with a convolution kernel size of 3×3, it is combined with the previous set of input features. After addition, the four sets of input features are input to the convolution layer of the corresponding path; the four sets of input features are aggregated in turn and input to their respective convolution layers to obtain four sets of intermediate feature results. Similarly, the bottom-up aggregation features are obtained

[0025] Finally, the intermediate feature results are input into two sets of 4×4 and 16×16 concatenated deconvolution layers to improve the feature resolution. The eight sets of intermediate feature results from the bidirectional path are concatenated to generate a tensor, which is then input into a stack consisting of three 3×3 convolutional layers and one 1×1 convolutional layer to obtain the pixel-level global feature map f g , expressed as:

[0026]

[0027] Furthermore, in step 2.2, the input visible light image I RGB The specific process of encoding local context information is as follows:

[0028] Step 2.2.1: Use a non-overlapping sliding window to RGB ∈R H×W×3 Sampling is performed, and the sliding window size is Image I RGB Decomposed into a sequence {I 1 ,I 2 ,I 3 ,I 4}; Each image sequence I n , n=1,2,3,4 decomposed into a sequence of fine-grained image blocks of size 8×8; then the image is mapped to the potential embedding space by linear projection, and the spatial position information corresponding to the original image is added to each image block to obtain the embedded variable

[0029] Step 2.2.2, embed variables Input local attention encoder L E , calculate the local attention information, the calculation expression is:

[0030]

[0031] Among them, the encoder L E The output feature sequence is expressed as Lr is the encoder L E Output number, set Lr = 12, C is the number of feature channels;

[0032] Step 2.2.3, similar to the global attention decoder G D , in the local attention decoder L using bidirectional multi-level aggregation D The feature sequence is upsampled and the output high-resolution feature sequences are merged;

[0033] First, the feature sequence Divide into four groups evenly and select the last feature in each group As decoder L D Input; after passing through the convolution layer with a convolution kernel size of 1×1, the feature map z l From the two-dimensional expression R Restore to three-dimensional expression R The number of channels remains unchanged;

[0034] Then, a bidirectional multi-stage aggregated local decoder L is used D Aggregate features from top to bottom and bottom to top respectively; and global attention decoder G D Differently, since the size of the image block sequence is smaller, all 3×3 convolutional layers are replaced with 1×1 convolutional layers to improve the accuracy of the edge; through the decoder L D , and obtain eight sets of intermediate feature results as well as

[0035] Finally, the intermediate feature results are input into two sets of 4×4 and 16×16 concatenated deconvolution layers to improve the feature resolution. The eight sets of intermediate feature results from the bidirectional path are concatenated to generate a tensor, which is then input into a stack consisting of three 3×3 convolution layers and one 1×1 convolution layer to obtain the pixel-level local feature map f. l , expressed as:

[0036]

[0037] Furthermore, in step 2.3, the edge prediction network L h The specific process of generating the final edge map is as follows:

[0038] Step 2.3.1, global feature map f g Processed by the feature modulation module: The feature modulation module contains a smoothing process composed of two 3×3 convolutional layers in series and an encoding operation composed of a deconvolution layer, a normalization layer, and an activation layer;

[0039] Step 2.3.2: The processed local feature map f l With the global feature map f g After concatenation and input into the feature modulation module, two 3×3 convolutional layers in series are used for smoothing again;

[0040] Step 2.3.3: Input the result of fusing global features and local features into the edge prediction network L h , and get the final edge graph I edge ; Among them, the edge prediction network L h Composed of a 1×1 convolutional layer and a sigmoid function, the generation of the edge map is expressed as:

[0041] Iedge =L h (FFM(f g ,f l )) (5).

[0042] Furthermore, in step 3, the original depth image is optimized using the visible light edge image to obtain the depth optimized image. The specific process is as follows:

[0043] Step 3.1: Transform the visible light edge image I edge and the original depth image I depth The spliced ​​image is represented as X={I edge ,I depth};

[0044] Step 3.2: Use the VGG-16 based multi-level feature extraction network as the encoder to extract the multi-level features of image X. Each level of features is input into the DenseASPP processing module; then, the five groups of features are spliced ​​according to the channel dimension;

[0045] Step 3.3: The resolution of the original depth image is reduced after the encoder extracts features. The feature map size is restored through four sets of decoders composed of 3×3 convolutional layers and bilinear upsampling layers in series, and a depth optimized image I′ with the same resolution as the original depth image is obtained. depth .

[0046] Furthermore, in step 4, a registration algorithm based on different coordinate system transformations is used to register the visible light edge image I edge and depth optimized image I′ depth The specific process of registration is as follows:

[0047] Step 4.1: The visible light camera and the depth camera are located on the same optical axis, and the lenses are placed on the same plane. Suppose there is a point P in space, and its imaging points on the visible light camera and the depth camera are P1 and P2 respectively. The relationship between image parallax and depth calculation is:

[0048]

[0049] Where Z is the depth of point P, f is the focal length of the camera, X L and X R are the distances from P1 and P2 to the left edge of the image, b is the camera baseline distance; let d = (X L -X R ), sorted as follows:

[0050]

[0051] Step 4.2: According to the camera imaging model, the transformation relationship between the pixel point (u, v) in the pixel coordinate system and the point P in the world coordinate system and its depth is obtained, which is expressed as:

[0052]

[0053] u0, v0 are the horizontal and vertical coordinates of the camera optical axis and the corner point of the image plane, dX, dY are the physical sizes of each pixel in the X-axis and Y-axis directions, R is the rotation matrix, T is the translation matrix, (X w ,Y w ,Z w ) is the three-dimensional position of point P in the world coordinate system; since there is no rotation and translation transformation, R, T can be simplified to the unit matrix and the all-zero column vector:

[0054]

[0055] According to the transformation relationship, we get:

[0056]

[0057] Step 4.3, in the depth camera, the depth camera coordinate system (X d ,Y d ,Z d ) and the depth image coordinate system (u d ,v d )The conversion relationship is:

[0058]

[0059] The above transformation process is equivalent to:

[0060]

[0061] So we get the depth camera coordinate system (X d ,Y d ,Z d ) and the depth image coordinate system (u d ,v d )Conversion relationship:

[0062]

[0063] Similarly, the visible light camera coordinate system (X c ,Y c ,Z c ) and the visible light image coordinate system (u c ,v c ) conversion relationship:

[0064]

[0065] Among them, M d and M c They are the depth camera intrinsic parameter matrix and the visible light camera intrinsic parameter matrix respectively. According to the conversion relationship between the camera coordinate system and the image coordinate system of the two cameras, the transformation relationship from the depth camera coordinate system to the visible light camera coordinate system is obtained:

[0066]

[0067] M is a 4×4 extrinsic parameter matrix. Substituting the transformation relationship between the visible light and depth camera coordinate systems and the image coordinate system, we get:

[0068]

[0069] According to the camera placement, the distance of the same point in the visible light camera and depth camera coordinate systems is the same, that is, Z d =Z c , so the formula is sorted as follows:

[0070]

[0071] in, is the parameter value of the matrix to be solved, and the conversion relationship from the depth image coordinate to the color image coordinate is obtained, and the registration is realized;

[0072]

[0073] Furthermore, in step 5, the specific process of detecting internal defects of the pipeline based on the visible light edge image and the depth optimization image and outputting a binary image of the detection result is as follows:

[0074] Step 5.1: Transform the visible light edge image I edge and depth optimized image I′ depth The spliced ​​image is represented by X′={I edge ,I′ depth};

[0075] Step 5.2: Use the salient feature encoder S E Extract the salient features S of image X′ d , random feature encoder R E Extract random features S of image X s ; for S s Multiple sampling is performed, and each pair of features After merging according to the channel dimension, multiple sets of intermediate feature results are obtained

[0076] In step 5.3, the intermediate feature results are passed through three sets of 3×3 convolutional layers and upsampling layers in series to output multiple sets of target detection prediction results. Then, based on the grayscale value statistics of each pixel in all the detection result images, the grayscale value that accounts for the majority of each pixel is selected to generate the final target detection result image.

[0077] Furthermore, in step 5.2, the specific process of extracting the significant features and random features of the image is as follows:

[0078] Step 5.2.1. Salient feature encoder S E The structure first extracts the multi-level features of X′, then inputs the features of each level into the DenseASPP processing module to expand the receptive field of the feature map and improve the feature expression ability. Finally, the five groups of features after processing are spliced ​​according to the channel dimension, and the number of feature map channels is modified using a 1×1 convolution layer to obtain the final salient feature S d ;

[0079] Step 5.2.2, random feature encoder R E Extract random features S of image X s First, through four sets of 3×3 convolutional layers in series and two sets of 3×3 average pooling layers in parallel, the two channels of the encoder respectively calculate the visible light edge image and the element depth image pair X = {I edge ,I depth}, which reflects the probability of the pixel corresponding to the target and background. Then, the constant ε that obeys the standard normal distribution is multiplied by σ and then the bias μ is added. Finally, the extracted features are filled to obtain the significant feature S. d Randomness characteristics S with consistent resolution s ;

[0080] Step 5.2.3, random feature S s Perform multiple sampling to obtain multiple sets of features Each set of random features and significant features are spliced ​​according to the channel dimension to obtain multiple sets of intermediate feature results

[0081] Furthermore, in step 6, based on the defect detection results, the pipeline internal defect classification network guided by deep saliency information is used to classify the defect detection results. The specific process is as follows:

[0082] Step 6.1. Set the resolution of the input image to 256×256.

[0083] Step 6.2: Use the encoding part V based on the VGG-16 network E,The feature extraction module consists of two 3×3 convolutional layers and a maximum pooling layer. The five feature extraction modules are connected in series to extract image features;

[0084] Step 6.3: Through two consecutive fully connected layers, the extracted image features are mapped to the sample space, which is set according to the defect type inside the pipeline. The Softmax function is used to convert all possible sample results into probabilities, as shown in the following formula:

[0085]

[0086] Among them, x i represents the i-th sample element, S i represents the Softmax result of the i-th sample element, and j represents the total number of samples. Through the Softmax function, the sample label with the highest probability corresponding to the image is obtained, and the final classification result is obtained.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] 1. This paper proposes a method for detecting internal defects in cable conduits using 3D information guided by saliency perception. This method effectively utilizes the 3D distance saliency information provided by depth images to guide target detection. By synergistically leveraging the information redundancy and complementarity of visible light and depth images, the accuracy of internal conduit defect detection is significantly improved.

[0089] 2. During the process of detecting internal pipeline image defects, this invention introduces an edge detection algorithm to extract edge information from the image. Based on the weak edge features of the pipeline itself, the contours of objects in the image are highlighted, enabling the identification of defects and foreign objects within the pipeline. By incorporating edge information from the image, the efficiency and accuracy of the algorithm's detection are improved.

[0090] 3. After detecting internal defects or foreign matter in pipelines, the present invention uses a pipeline defect classification method guided by depth saliency information to accurately distinguish different types of defects based on the depth saliency and edge characteristics of the defects. This allows technicians to propose targeted solutions based on the defect type.

[0091] 4. The saliency perception-guided internal defect detection method for cable conduits proposed in the present invention can still correctly detect defects and their categories even when the lighting conditions inside the conduit are poor and the defects are highly similar to the background. The algorithm has high precision and accuracy and can be applied to conduit laying, cable threading and subsequent conduit maintenance work, with a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Flow chart of the method of the present invention;

[0093] Figure 2This is the flow chart of the EDTER edge detection algorithm;

[0094] Figure 3 It is a multi-level feature aggregation flow chart;

[0095] Figure 4 This is a diagram of the multi-level feature extraction network structure based on VGG-16;

[0096] Figure 5 Diagram of the network encoder structure for pipeline internal defect classification guided by deep salient information. DETAILED DESCRIPTION

[0097] The specific implementation of the cable conduit internal defect detection method guided by significant perception of the present invention is described in detail below with reference to the accompanying drawings.

[0098] The visual information acquisition component of the proposed saliency-sensing-guided cable conduit internal defect detection method consists of a visible light camera and a structured light-based depth camera. Both the visible light image and the depth image have a resolution of 1920 × 1080. The cameras are aligned on the same optical axis, the lenses are placed in the same plane, and the baseline distance is 120 mm.

[0099] like Figure 1 The specific process of the cable conduit internal defect detection method guided by significant perception proposed in the present invention is as follows:

[0100] Step 1. Use a visible light camera and a depth camera based on structured light to obtain visible light images of the interior of the pipeline. RGB and depth image I depth .

[0101] Step 2: Use the EDTER edge detection algorithm to extract the edge information of the visible light image and obtain the visible light edge image I edge The EDTER edge detection algorithm flow chart is as follows: Figure 2 As shown, the visible light image I RGB The size is H×W, the number of channels is 3, and the image can be represented as I RGB ∈R H×W×3 The specific process of extracting visible light edge images is as follows:

[0102] Step 2.1. Input visible light image I RGB Encode global context information. First, the input image is segmented into a series of coarse-grained image sequences and a global attention encoder is used to learn global context information. Then, a global bidirectional multi-stage aggregation decoder is used to generate a global feature map.

[0103] Step 2.1.1. First, evenly divide the image into two-dimensional images I RGB ∈R H×W×3Convert it into a one-dimensional image block sequence to obtain a coarse-grained image block sequence of size 16×16. Then, through linear projection, the one-dimensional image block sequence is mapped to the potential embedding space, and the spatial position information corresponding to the original image is added to each image block to obtain the embedded variable

[0104] Step 2.1.2. Embed variables Input global attention encoder G E , calculate the global attention information. The calculation expression is as follows:

[0105]

[0106] Among them, the encoder G E The output feature sequence is expressed as Lg is the encoder G E Output number, C is the number of feature channels. Specifically, set Lg = 24;

[0107] Step 2.1.3. Transform the feature sequence z g Input is a global attention decoder G with bidirectional multi-level aggregation D Perform upsampling to obtain a high-resolution feature decoder. D A bidirectional feature aggregation strategy is used, including a top-down path that fuses low-level features from high-level features and a bottom-up path that fuses high-level features from low-level features;

[0108] First, the feature sequence Divide into four groups and take the last feature of each feature sequence As decoder G D After passing through the convolution layer with a convolution kernel size of 1×1, the feature map z g From the two-dimensional expression R Restore to three-dimensional expression R The number of channels remains unchanged;

[0109] Then, features are aggregated from top to bottom. Figure 3 shown, specifically, After inputting into the convolution layer with a convolution kernel size of 3×3, it is combined with the previous set of input features. After adding, the four sets of input features are input to the convolution layer of the corresponding path. The four sets of input features are aggregated in sequence and input to their respective convolution layers to obtain four sets of intermediate feature results. Similarly, the bottom-up aggregation features are obtained

[0110] Finally, the intermediate feature results are input into two sets of 4×4 and 16×16 concatenated deconvolution layers to improve feature resolution. After the eight sets of intermediate feature results from the bidirectional path are concatenated to generate a tensor, they are input into a stack consisting of three 3×3 convolution layers and one 1×1 convolution layer to obtain the pixel-level global feature map f g , expressed as:

[0111]

[0112] Step 2.2. Input visible light image I RGB Perform local context encoding. Similar to the global context encoding process, the input image is first sampled using a non-overlapping sliding window, dividing the entire image into multiple fine-grained image block sequences. A local attention encoder is then executed on each image block sequence to capture short-range local cues. Finally, all features are input into a local bidirectional multi-stage aggregation decoder to generate a local feature map.

[0113] Step 2.2.1. Using non-overlapping sliding windows to map the visible light image I RGB ∈R H×W×3 Sampling is performed, and the sliding window size is Image I RGB Decomposed into a sequence {I 1 ,I 2 ,I 3 ,I 4}. Similar to the global context coding method, each image sequence I n ,n=1,2,3,4 decomposed into a sequence of fine-grained image blocks of size 8×8. Then the image is mapped to the potential embedding space by linear projection, and the spatial position information corresponding to the original image is added to each image block to obtain the embedded variable

[0114] Step 2.2.2. Embed variables Input local attention encoder L E , calculate the local attention information. The calculation expression is as follows:

[0115]

[0116] Among them, the encoder L E The output feature sequence is expressed as Lr is the encoder L E Output number, C is the number of feature channels. Specifically, set Lr = 12;

[0117] Step 2.2.3. Similar to the global attention decoder G D, in the local attention decoder L using bidirectional multi-level aggregation D The feature sequence is upsampled and the output high-resolution feature sequences are merged;

[0118] First, the feature sequence is still Divide into four groups evenly and select the last feature in each group As decoder L D After passing through the convolution layer with a convolution kernel size of 1×1, the feature map z l From the two-dimensional expression R Restore to three-dimensional expression R The number of channels remains unchanged;

[0119] Then, a bidirectional multi-stage aggregated local decoder L is used D Aggregate features from top to bottom and bottom to top respectively. With global attention decoder G D Differently, since the size of the image block sequence is smaller, all 3×3 convolutional layers are replaced with 1×1 convolutional layers to improve the accuracy of the edge. D , and obtain eight sets of intermediate feature results as well as

[0120] Finally, the intermediate feature results are input into two sets of 4×4 and 16×16 concatenated deconvolution layers to improve feature resolution. After the eight sets of intermediate feature results from the bidirectional path are concatenated to generate a tensor, they are input into a stack consisting of three 3×3 convolution layers and one 1×1 convolution layer to obtain the pixel-level local feature map f l , expressed as:

[0121]

[0122] Step 2.3. Global feature map f g and local feature f l It is processed by the feature fusion module (FFM) and then input into the edge prediction network L h To generate the final edge map;

[0123] Step 2.3.1. Global feature map f g Processed by a feature modulation module. The feature modulation module includes a smoothing operation consisting of two 3×3 convolutional layers in series and an encoding operation consisting of a deconvolution layer, a normalization layer, and an activation layer;

[0124] Step 2.3.2. The processed local feature map f l With the global feature map f g After concatenation and input into the feature modulation module, two 3×3 convolutional layers in series are used for smoothing again;

[0125] Step 2.3.3. Input the result of fusing global features and local features into the edge prediction network L h , and get the final edge graph I edge Among them, the edge prediction network L h It consists of a 1×1 convolutional layer and a sigmoid function. The generation of the edge map can be expressed as:

[0126] I edge =L h (FFM(f g ,f l ))(5).

[0127] Step 3. Use the visible light edge image to optimize the original depth image to obtain the depth optimized image I′ depth ;

[0128] Step 3.1. Transform the visible light edge image I edge and the original depth image I depth The spliced ​​image is represented as X={I edge ,I depth};

[0129] Step 3.2. Use a multi-level feature extraction network based on VGG-16 as the encoder, such as Figure 4 As shown. Extract multi-level features of image X Each level of features is input into the DenseASPP processing module to expand the receptive field of the feature map and improve the feature expression ability. Then, the five groups of features are spliced ​​according to the channel dimension;

[0130] Step 3.3. The resolution of the original depth image is reduced after the encoder extracts features, and the feature map size is restored through a decoder composed of four sets of 3×3 convolutional layers and bilinear upsampling layers in series. The depth optimized image I′ with the same resolution as the original depth map is obtained. depth .

[0131] Step 4. Use the registration algorithm based on different coordinate system transformation to register the visible light edge image I edge and depth optimized image I′ depth Registration;

[0132] Step 4.1. The visible light camera and the depth camera are located on the same optical axis, and the lenses are placed on the same plane. Suppose there is a point P in space, and its image points on the visible light camera and the depth camera are P1 and P2 respectively. The relationship between image disparity and depth calculation is as follows:

[0133]

[0134] Where Z is the depth of point P, f is the focal length of the camera, X L and X R are the distances from P1 and P2 to the left edge of the image, b is the camera baseline distance 120 mm, let d = (X L -X R ), we can get:

[0135]

[0136] Step 4.2. According to the camera imaging model, the transformation relationship between the pixel point (u, v) in the pixel coordinate system and the point P in the world coordinate system and its depth is obtained, which is expressed as:

[0137]

[0138] u0, v0 are the horizontal and vertical coordinates of the camera optical axis and the corner point of the image plane, dX, dY are the physical sizes of each pixel in the X-axis and Y-axis directions, R is the rotation matrix, T is the translation matrix, (X w ,Y w ,Z w ) is the three-dimensional position of point P in the world coordinate system; since there is no rotation and translation transformation, R, T can be simplified to the unit matrix and the all-zero column vector:

[0139]

[0140] According to the transformation relationship, we can get:

[0141]

[0142] Step 4.3. In the depth camera, the depth camera coordinate system (X d ,Y d ,Z d ) and the depth image coordinate system (u d ,v d )The conversion relationship is:

[0143]

[0144] The above transformation process is equivalent to:

[0145]

[0146] So we get the depth camera coordinate system (X d ,Y d ,Z d ) and the depth image coordinate system (u d ,v d )Conversion relationship:

[0147]

[0148] Similarly, the visible light camera coordinate system (X c ,Y c ,Z c ) and the visible light image coordinate system (u c ,v c ) conversion relationship:

[0149]

[0150] Among them, M d and M c They are the depth camera intrinsic parameter matrix and the visible light camera intrinsic parameter matrix respectively. According to the conversion relationship between the camera coordinate system and the image coordinate system of the two cameras, the conversion relationship from the depth camera coordinate system to the visible light camera coordinate system can be obtained:

[0151]

[0152] M is a 4×4 extrinsic parameter matrix. Substituting the transformation relationship between the visible light and depth camera coordinate systems and the image coordinate system, we can obtain:

[0153]

[0154] According to the camera placement, the distance of the same point in the visible light camera and depth camera coordinate systems is the same, that is, Z d =Z c . Therefore, the formula can be organized as:

[0155]

[0156] in, is the parameter value of the matrix to be solved. From this, the conversion relationship from the depth image coordinates to the color image coordinates is obtained, and registration is achieved;

[0157]

[0158] Step 5. The pipeline internal defect detection network based on RGB-D information collaboration utilizes the complementary information of visible light images and depth images to detect internal defects in the pipeline based on the visible light edge image and depth-optimized image. The network then outputs a binary image of the detection results, achieving accurate detection of internal defects in the pipeline.

[0159] Step 5.1. Transform the visible light edge image I edge and depth optimized image I′ depth The spliced ​​image is represented by X′={I edge ,I′ depth};

[0160] Step 5.2. Use the salient feature encoder S respectively E Extract the salient features S of image X′ d , random feature encoder R E Extract random features S of image X s . s Multiple sampling is performed, and each pair of features After merging according to the channel dimension, multiple sets of intermediate feature results are obtained

[0161] Step 5.2.1. Salient feature encoder S E The structure is similar to the multi-level feature extraction network based on VGG-16. First, the multi-level features of X′ are extracted. Then, the features of each level are input into the DenseASPP processing module to expand the receptive field of the feature map and improve the feature expression ability. Finally, the five groups of processed features are spliced ​​according to the channel dimension, and the number of feature map channels is modified using a 1×1 convolution layer to obtain the final saliency feature S. d ;

[0162] Step 5.2.2. Random feature encoder R E Extract random features S of image X s First, through four sets of 3×3 convolutional layers in series and two sets of 3×3 average pooling layers in parallel, the two channels of the encoder respectively calculate the visible light edge image and the element depth image pair X = {I edge ,I depth The probability distribution parameters μ and σ of} reflect the probability of the pixel corresponding to the target and background. Then, the constant ε that obeys the standard normal distribution is multiplied by σ and then added with the bias μ; finally, the extracted features are filled to obtain the salient feature S d Randomness characteristics S with consistent resolution s ;

[0163] Step 5.2.3. For random feature S s Perform multiple sampling to obtain multiple sets of features Each set of random features and significant features are spliced ​​according to the channel dimension to obtain multiple sets of intermediate feature results

[0164] Step 5.3: The intermediate feature results are passed through three sets of 3×3 convolutional layers and upsampling layers in series, outputting multiple sets of object detection prediction results. Then, based on the grayscale value statistics of each pixel in all detection result images, the grayscale values ​​that account for the majority of pixels are selected to generate the final object detection result image.

[0165] Step 6. Based on the defect detection results, the pipeline internal defect classification network guided by deep saliency information is used to classify the defect detection results and correctly identify the internal defect category of the pipeline;

[0166] Step 6.1. First, reduce the input image resolution from 1920×1080 to 256×256. This significantly increases the algorithm's speed without sacrificing classification accuracy.

[0167] Step 6.2. Then, use the encoding part V based on the VGG-16 network E ,like Figure 5 As shown in the figure, a feature extraction module consisting of two 3×3 convolutional layers and a maximum pooling layer was designed. The five feature extraction modules were connected in series to extract image features. The number of channels in the convolutional layers of the five modules was set to 64, 128, 256, 512, and 512, respectively. By extracting deep salient features of defect targets, more accurate classification results can be provided.

[0168] Step 6.3. Finally, through two consecutive fully connected layers, the extracted abstract distributed features are mapped to the sample space, which is set according to the defect type inside the pipeline. All possible sample results are converted into probabilities through the Softmax function, as shown in the following formula:

[0169]

[0170] Among them, x i represents the i-th sample element, S i represents the Softmax result of the i-th sample element, and j represents the total number of samples. Through the Softmax function, the sample label with the highest probability corresponding to the image is obtained, and the final classification result is obtained.

[0171] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit the present invention. Those skilled in the art will appreciate that they may modify or substitute equivalents for the specific embodiments of the present invention, provided that such modifications or variations are within the scope of protection of the pending claims.

Claims

1. A method for detecting internal defects of cable conduits guided by significant perception, characterized in that: The method comprises the following steps: Step 1: Use a visible light camera and a structured light-based depth camera to obtain visible light images of the interior of the pipeline. and depth image ; Step 2: Extract visible light image using EDTER edge detection algorithm The edge information of the visible light edge image is obtained ; Step 3: Use visible light edge image For depth images Optimize to get a deep optimized image ; Step 4: Use the registration algorithm based on different coordinate system transformations to register the visible light edge image. and deeply optimized images Registration; Step 5: Based on the visible light edge image and deeply optimized images Detect internal defects in pipelines and output binary images of detection results; Step 6: Based on the defect detection results, the pipeline internal defect classification network guided by deep saliency information is used to classify the defect detection results and correctly identify the internal defect category of the pipeline; In step 2, the edge information of the visible light image is extracted using the EDTER edge detection algorithm. The specific process of obtaining the visible light edge image is as follows: Step 2.1: Input visible light image Encode global context information: First, the input image is divided into a series of coarse-grained image sequences, and a global attention encoder is used to learn global context information; then a global bidirectional multi-level aggregation decoder is used to generate a global feature map ; Step 2.2: Input visible light image Encode local context information: First, use a non-overlapping sliding window to sample the input image and divide the entire image into multiple fine-grained image block sequences; then perform a local attention encoder on each image block sequence to capture short-range local clues; finally, all features are input into a local bidirectional multi-stage aggregation decoder to generate a local feature map ; Step 2.3, global feature map and local features Processed by the feature fusion module FFM and then input into the edge prediction network To generate visible light edge images ; In step 3, the visible light edge image is used For depth images Optimize to get a deep optimized image The specific process is: Step 3.1: Visible light edge image and the original depth image The spliced ​​image is represented as follows according to the channel dimension: ; Step 3.2: Use the VGG-16 based multi-level feature extraction network as the encoder to extract the image Each level of features is input into a densely connected spatial pyramid pooling processing module; then, the five groups of features are spliced ​​according to the channel dimension; Step 3.3: Original depth image The resolution is reduced after the encoder extracts the features, and the The decoder of the convolution layer and the bilinear upsampling layer restores the feature map size and obtains the same depth image as the original Deeply optimized images with consistent resolution .

Images