Pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereoscopic vision

Abstract

The invention relates to the field of pipeline three-dimensional reconstruction and intelligent detection, in particular to a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereoscopic vision. According to the technical scheme, a panoramic camera and a radar sensor are used for collecting panoramic images and depth data in a pipeline; carrying out denoising, distortion correction and illumination compensation processing on the collected panoramic image, and carrying out image optimization; stable feature points in the panoramic image are extracted, and feature point matching is carried out; a multi-view stereoscopic vision algorithm and a synchronous positioning and mapping technology are combined, and pose estimation and three-dimensional point cloud generation are carried out by using a panoramic image and depth data; optimizing the generated point cloud data, and reconstructing a smooth and continuous three-dimensional surface; and based on the processed three-dimensional data, intelligently detecting cracks and corrosion defects in the pipeline by using a convolutional neural network, and automatically generating defect types, positions and repair suggestions. The method is suitable for pipeline detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pipeline three-dimensional reconstruction and intelligent detection, and particularly relates to a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision. BACKGROUND

[0002] From the perspective of technical implementation, in the aspect of pipeline detection, the existing technology has significant shortcomings in multi-modal sensor fusion, adaptability to complex environments, and intelligent defect analysis. In terms of multi-sensor data synchronization, existing solutions mostly use discrete sensor configurations, lacking an efficient spatiotemporal calibration mechanism, which leads to cumulative registration errors of image and depth information, especially in dynamic deformation scenarios of pipelines, making it difficult to meet the demand for high-precision reconstruction. Feature matching in low-texture areas is another technical difficulty, as traditional algorithms fail in smooth metal pipe walls or dirt-covered surfaces, resulting in loss of key frames or a sharp increase in mismatching rates in three-dimensional reconstruction. The real-time challenge in dynamic environments is due to the high computational complexity of SLAM (Simultaneous Localization and Mapping) algorithms in complex topologies. When there are a large number of branch structures or obstacles in the pipeline, the system often falls into a state of local convergence or tracking loss. In addition, existing defect detection models are mostly based on shallow feature extraction, which is sensitive to changes in lighting and noise interference, and lacks deep understanding of the development patterns of pipeline diseases, resulting in high false negative and false positive rates. These technical bottlenecks make it difficult for existing systems to meet the urgent needs of modern urban pipeline management, and breakthroughs are needed through multidisciplinary cross-fusion and independent innovation.

[0003] In summary, the existing detection technology has the following pain points: Pain point one: multi-modal sensor data fusion failure. The spatiotemporal calibration error of optical cameras and lidar in existing solutions easily leads to data registration deviation, and in dynamic deformation scenarios of pipelines, the relative pose drift between sensors can cause three-dimensional reconstruction to break, and it is difficult to achieve cross-modal feature complementation in low-texture areas.

[0004] Pain point two: insufficient low-texture environment perception capability. Traditional feature extraction algorithms cannot stably match in smooth metal pipe walls or dirt-covered surfaces, resulting in loss of key frames and trajectory jumps, and cannot meet the demand for high-precision pose estimation in complex scenarios such as pipeline bending and rusting.

[0005] Pain point three: poor real-time processing and adaptability to complex topologies. The existing SLAM algorithm has high computational complexity in complex topologies such as pipeline branching and diameter reduction, resulting in a system response delay of more than 2 seconds per frame, and cannot balance reconstruction accuracy and efficiency through incremental optimization, restricting the application feasibility of dynamic monitoring scenarios. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art, provide a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision, which realizes high-precision three-dimensional modeling and intelligent defect diagnosis in the whole space range of the pipeline.

[0007] The present application achieves the above-mentioned purpose by adopting the following technical solutions, and provides a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision, comprising: S1, using a panoramic camera and a radar sensor to collect panoramic images and depth data inside the pipeline; S2, denoising, distortion correction and illumination compensation processing are performed on the collected panoramic images, and image optimization is performed; S3, using SuperPoint feature extraction algorithm to extract stable feature points in the panoramic image, and using SuperGlue algorithm for feature point matching; S4, combining multi-view stereo vision algorithm and SLAM (Simultaneous Localization and Mapping, simultaneous localization and mapping) technology, using panoramic images and depth data for pose estimation and three-dimensional point cloud generation; S5, the generated point cloud data is optimized by incremental Bundle Adjustment algorithm, and the improved Poisson Surface Reconstruction algorithm is used to reconstruct smooth and continuous three-dimensional surface; S6, based on the three-dimensional data processed in step S5, using convolutional neural network to intelligently detect the cracks and corrosion defects inside the pipeline, and automatically generating defect type, location and repair suggestion; S7, through cloud platform storage and management of image and three-dimensional point cloud data, combining augmented reality or virtual reality technology to visualize the data, and supporting remote access and real-time monitoring.

[0008] Further, step S1 specifically comprises: The panoramic camera is equipped with 8 groups of 1 / 2.3 inch CMOS sensor, supports 5760x2880 ultra-high definition resolution, is equipped with F2.8 large aperture fisheye lens, and is built-in gyro and accelerometer to assist pose solution, through dynamic exposure adjustment to adapt to 0.1 lux-100klux wide dynamic light environment, the radar sensor is configured with 360° horizontal scanning range and 0.1° vertical resolution, through GPIO interface and panoramic camera to realize nanosecond level time synchronization, equipped with multi-echo detection technology to distinguish liquid, solid sediment and metal pipe wall reflection signal, and configured with waterproof aviation joint to adapt to wet environment, the radar component is installed at the end of the mechanical arm, and the camera forms a complementary observation angle at a 90° angle.

[0009] Further, step S2 specifically comprises: Denoising the collected panoramic image using a DnCNN deep learning model, the DnCNN deep learning model adopts a convolutional neural network structure to remove Gaussian noise and salt and pepper noise in the panoramic image; Geometric correction is performed on the collected panoramic image through a distortion correction algorithm to remove radial and tangential distortion caused by the fisheye lens and restore the true geometric shape of the panoramic image; An adaptive light compensation algorithm is used to optimize the brightness and contrast of the image. In low light conditions, the exposure and brightness are automatically adjusted to enhance the visibility of the image. Local brightness enhancement technology is used to enhance the local brightness of the installed area in the pipeline. In combination with image cropping technology, the panoramic image is cropped to focus on the installed part of the pipeline. After cropping, an image enhancement algorithm is used to improve the contrast and clarity of the panoramic image, highlighting the installed part information in the panoramic image.

[0010] Further, step S3 specifically comprises: Using the SuperPoint feature extraction algorithm to automatically extract stable feature points from the panoramic image of the pipeline; Using the SuperGlue algorithm to match the feature points extracted under different viewing angles to ensure accurate alignment between images; Remove matching points that do not meet geometric constraints through the random sample consensus algorithm, and then optimize the matched feature points and camera pose through the BundleAdjustment algorithm.

[0011] Further, step S4 specifically comprises: Use multi-view stereo vision algorithm to estimate depth from panoramic images from different viewing angles. Through pixel alignment and reconstruction of multi-view images, high-density three-dimensional point cloud data is generated; In combination with SLAM technology, the pose of the panoramic image data is estimated. Through the combination of SLAM technology and depth information, the accurate calculation of the motion trajectory of the panoramic camera in the pipeline is realized. In the real-time pose tracking link, the ORB-SLAM3 algorithm is used to construct the topological relationship between local and global maps. Through ORB (Oriented FAST and Rotated BRIEF) feature extraction and matching, the motion relationship between key frames is established. Combined with IMU (IMU Pre-integration Constraint) pre-integration constraint, the robustness of pose estimation in dynamic scenes is improved. Kalman filter is introduced to predict the camera motion trajectory, and ICP point cloud registration algorithm is used for real-time correction; On the basis of MVS and SLAM algorithms, through optimization of camera internal and external parameters, errors are eliminated, and the accuracy of the three-dimensional point cloud is improved. The three-dimensional information of the area with insufficient illumination or missing texture in the pipeline is filled by using the depth data collected by the radar sensor. An incremental three-dimensional reconstruction method is adopted, new images and depth data are continuously added at each collection point, and three-dimensional reconstruction is gradually completed.

[0012] Further, step S5 specifically comprises: The incremental Bundle Adjustment algorithm is used for global optimization of the three-dimensional point cloud and the camera pose, the incremental Bundle Adjustment algorithm continuously optimizes the matching accuracy of the camera pose and the three-dimensional point cloud by minimizing the geometric error between each feature point in the panoramic image and the camera, and ensures the maximum consistency of all collected panoramic images and point cloud data in the pipeline; The spatial position of the entire point cloud model is optimized through the graph optimization technology, the spatial constraint of the point cloud and the panoramic camera is adjusted, and the global consistency of the three-dimensional reconstruction is ensured; The Poisson Surface Reconstruction algorithm is adopted to calculate the surface normal from the sparse point cloud, and then a smooth surface is generated; The surface reconstruction effect is optimized by combining the multi-resolution technology, and for the details of the complex area, the reconstruction is refined by the multi-resolution method; Through the depth compensation algorithm, the depth data of the low-texture, unevenly illuminated or occluded area is supplemented.

[0013] The beneficial effects of the present application are: The present application adopts the space-time synchronous collection mechanism of the panoramic camera and the laser radar, solves the feature matching failure problem in the low-texture area through the dynamic error compensation algorithm, and can still stably acquire three-dimensional data even in the pipeline bending or dirt covered scene; the Retinex photometric distortion correction and adaptive exposure control technology are introduced, stable feature extraction is realized under 0.1 lux extremely weak illumination, and the image registration accuracy in a complex scene is effectively improved in combination with the feature matching algorithm. The application fuses MVS-SLAM3 and ICP point cloud registration technology, introduces Kalman filter to predict the camera motion state, and significantly improves the reconstruction stability of complex topological structure; a multi-scale deep learning model is constructed to realize pixel-level segmentation and three-dimensional quantitative analysis of pipeline diseases, and provide high reliability basis for defect detection; finally, remote operation and maintenance management and scientific decision-making are realized through an augmented reality visualization platform and an intelligent evaluation model, and the pipe network maintenance efficiency is improved. On the hardware level, the system integrates panoramic cameras, laser radars and mechanical arms, and the data processing module relies on a high-performance computing platform to complete real-time feature extraction and three-dimensional reconstruction, and cloud management realizes data storage, analysis and decision support through the Internet of Things and a visualization engine. The application breaks through the environmental adaptability bottleneck of traditional detection equipment, and provides an efficient and reliable intelligent solution for fine operation and maintenance of urban infrastructure. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a pipeline three-dimensional reconstruction and intelligent detection method flowchart based on panoramic stereo vision provided by an embodiment of the application. DETAILED DESCRIPTION

[0015] To make the purpose, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0016] The application provides a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision, as shown in Figure 1 , specifically comprising: S1, using a panoramic camera and a radar sensor to collect panoramic images and depth data in a pipeline; Specifically, the application uses a panoramic camera and a radar sensor to synchronously collect panoramic images and depth data in a pipeline. The panoramic camera is equipped with 8 groups of 1 / 2.3 inch CMOS sensors, supports 5760x2880 ultra-high definition resolution (equivalent to 28 million pixels of a monocular camera), is equipped with a F2.8 large-aperture fisheye lens (horizontal / vertical field of view 180°), and is internally provided with a gyroscope (±0.5° / s precision) and an accelerometer (±2g precision) to assist in pose solving. The panoramic camera can adapt to a wide dynamic light environment of 0.1 lux-100 klux through dynamic exposure adjustment (ISO 100-3200 adjustable, shutter speed 1 / 30s-1 / 1000s), and can realize seamless splicing at a pipeline bending place by enabling a FlowState anti-shake algorithm. The device is directly connected to a data processing module through a gigabit Ethernet, the data transmission delay is controlled within 50 ms, and 5G / LoRa dual-mode wireless transmission (breakpoint resume function) is supported.

[0017] The radar sensor is configured with a 360° horizontal scanning range and a 0° vertical resolution, a ranging accuracy of ±2 mm at 10 m, a scanning frequency of 20 Hz, and a point cloud density of 120-1200 pts / m 2 Adjustable. The radar achieves nanosecond-level time synchronization with the panoramic camera through a GPIO interface, supports multi-echo detection technology to distinguish liquid (sewage), solid deposits (silt) and metal pipe wall reflection signals, and is equipped with a waterproof aviation joint (IP68 protection level) to adapt to wet environments. The radar component is installed at the end of the mechanical arm, forming a complementary observation angle of 90° with the camera. In the dynamic obstacle avoidance mode, the scanning frequency is increased to 30 Hz.

[0018] To realize multi-modal data space-time registration, the present application uses a checkerboard calibration board (containing infrared reflective marker points) to complete camera-radar external parameter calibration (translation error ≤0.3 mm, rotation error ≤0.05°), and develops a distortion correction model based on deep learning (U-Net architecture) to optimize the radial distortion coefficients k1, k2 to within 0.001. Through the message publishing / subscription mechanism in the ROS middleware, combined with the GPS timing module (±1 μs accuracy), the timestamps of the image and point cloud data are synchronized, ensuring that their frame numbers are strictly aligned. This synchronization mechanism supports multi-sensor data fusion at the pipe bifurcation, effectively avoiding the SLAM trajectory splitting problem.

[0019] S2, denoising, distortion correction and illumination compensation processing are performed on the collected panoramic image, and image optimization is performed; The DnCNN deep learning model is used to denoise the collected panoramic image, the noise features are extracted through the convolution layer, and the training data set contains low-light pipe images, which effectively suppresses Gaussian noise and salt and pepper noise; Based on the Retinex photometric distortion correction theory, combined with adaptive filtering to separate incident light and reflected light components, the distortion correction algorithm is used to correct the collected panoramic image, remove the radial and tangential distortion caused by the fisheye lens, and restore the true geometric shape of the panoramic image; The specific steps are as follows: Retinex light field decomposition is used to accept denoised images to increase image dark area details; Geometric distortion correction is performed by loading calibration parameters for fisheye correction, and special pipe sections are compensated to reduce distortion.

[0020] An adaptive illumination compensation algorithm is used to optimize the brightness and contrast of the image. In low light conditions, the exposure and brightness are automatically adjusted to enhance the visibility of the image, and local enhancement technology is used to enhance the local brightness of the pipe installation area; Combined with image cropping technology, the panoramic image is cropped to focus on the setting part of the pipeline. After cropping, the image enhancement algorithm is used to improve the contrast and clarity of the panoramic image, highlighting the setting part information in the panoramic image.

[0021] The spatial downsampling technique is used to compress the point cloud data volume, and the voxel size is set to a fixed resolution, which significantly reduces the subsequent computational complexity. Abnormal points are removed by calculating the density of neighboring points, and the effective geometric features of the inner wall of the pipeline are preserved. The iterative closest point algorithm is used to optimize the mapping relationship between the point cloud and the image pixels, and the convergence threshold is set to realize the rigid body transformation. The pixel-point cloud projection matrix is established, and the depth information is mapped to the image pixel level through perspective transformation to generate an enhanced image with three-dimensional coordinate labeling, supporting subsequent feature matching and defect positioning.

[0022] S3, use SuperPoint feature extraction algorithm to extract stable feature points in panoramic image, and use SuperGlue algorithm for feature point matching to ensure accurate alignment and high-quality matching of images in low light and low texture environment; Use SuperPoint feature extraction algorithm to automatically extract stable feature points from pipeline images. SuperPoint algorithm can effectively identify stable feature points in areas with scarce texture or insufficient lighting inside the pipeline. Based on convolutional neural network architecture, it can automatically extract high-quality feature points in low light and complex backgrounds, and generate corresponding 128-dimensional descriptors, adapting to challenges such as deformation and lighting changes in different pipeline environments.

[0023] The specific steps of Superpoint feature point extraction are as follows: Image standardization preprocessing: accept the corrected panoramic image, normalize the image, and output a standard grayscale image; Lightweight model inference: divide the image grid, output 128-dimensional descriptors, generate a feature point set, and a descriptor matrix; Feature optimization enhancement: through non-maximum suppression, retain feature points with a confidence level ≥0.85.

[0024] Use SuperGlue algorithm for feature point matching to efficiently match feature points extracted under different perspectives, ensuring accurate alignment between images. SuperGlue algorithm optimizes the association between feature points by introducing graph neural networks, which can accurately match feature points even in the presence of occlusions, lighting changes, and texture deficiencies in pipeline images, reducing the likelihood of false matches.

[0025] The specific steps of SuperGlue feature matching are as follows: Construct feature association: receive the feature point coordinates and descriptor matrix output by superpoint, and create a bidirectional graph structure; Attention mechanism matching: calculate the descriptor similarity matrix, and perform multi-head attention calculation to generate the matching confidence matrix; Matching screening: use Sinkhorn algorithm for optimal transport solution, and verify consistency.

[0026] In combination with the RANSAC algorithm, the false matching is removed, and the random sampling consistency algorithm is used to remove the matching points that do not meet the geometric constraints, so that the high quality and accuracy of the remaining feature points are ensured. This step can automatically identify and remove false matching points caused by noise, occlusion or other external factors in the pipeline image, and optimize the feature matching result.

[0027] In order to enhance the image features in low light and low texture environment, the present application improves the robustness of feature matching by combining deep learning with classical algorithms. Specifically, the 256x256 resolution preprocessed image is received at the output layer, and the Retinex theory model is used to separate the light component L(x,y) and the reflection component R(x,y), which is mathematically expressed as: I(x,y)=L(x,y)⋅R(x,y)+ϵ.

[0028] Wherein, epsilon is a noise term. By suppressing the reflection component R(x,y), the interference of the metal pipe wall reflection can be effectively reduced.

[0029] In the training stage, an adversarial generative network is introduced, and its adversarial loss function is defined as: a feature extraction network containing 8 convolutional layers is constructed, each layer is configured with a 3x3 convolution kernel, batch normalization and LeakyReLU activation function, and the last layer outputs a 128-dimensional feature descriptor and is processed by L2 normalization to improve the robustness of cross-frame matching. In order to solve the problem of lack of low-texture data of real pipelines, the present application designs a conditional generative adversarial network training framework, which includes a generator and a discriminator: the generator takes random noise and pipeline category label as input, and outputs low-texture synthetic images simulating defects such as oil stains, rust and scratches; the discriminator needs to distinguish between real images and generated images, and predict the pipeline category of the image. By jointly optimizing the adversarial loss, feature matching loss and classification loss, the generated image has both authenticity and diversity.

[0030] In order to realize feature extraction of complex curved surfaces (such as pipeline elbows and reduced diameter), the present application introduces a multi-scale feature pyramid mechanism: the down-sampling branch gradually compresses the spatial dimension through convolution layers with a step of 2, and captures global contour features; the up-sampling branch combines detail information through transpose convolution and jump connection, and finally dynamically adjusts the fusion ratio of features of different scales through an adaptive weight distribution module - this module distributes weights according to the local image gradient intensity, and suppresses the noise interference in low-texture areas.

[0031] To meet the real-time requirements of pipeline detection (single-frame processing ≤120 ms), the present application adopts a dual strategy of hardware acceleration and algorithm optimization: quantization and pruning of the network, which increases the model inference speed by 3 times; through sliding window detection, the panoramic image is divided into 16x16 pixel local blocks, and the features of each window are calculated in parallel to improve the throughput; at the same time, the original SuperPoint network is compressed from 32 layers of convolution to 8 layers, reducing the parameter amount by 60% while maintaining mAP ≥ 90%.

[0032] In view of the feature failure problem caused by the mirror reflection of the metal pipe wall, the present application integrates photometric distortion compensation: a Retinex-Net network is used to estimate the incident light intensity map and the reflectivity map of the reflected light in real time, and the image color balance is dynamically adjusted to suppress the highlight overflow artifact; the context information is used to fill the content of the shadow area, and the semantic segmentation network is used to identify the oil stain covered area and to enhance the texture details accordingly.

[0033] S4, combined with multi-view stereo vision algorithm and SLAM technology, panoramic image and depth data are used for pose estimation and three-dimensional point cloud generation; By combining simultaneous localization and mapping (SLAM) technology, the image data is subjected to pose estimation, and the combination of image and depth information is optimized by SLAM to realize accurate calculation of the camera motion trajectory in the pipeline. SLAM technology can accurately estimate the motion trajectory of the camera and the position relationship in the three-dimensional space in an unknown environment, solving the problem of dynamic environment change that traditional methods cannot handle, and ensuring that each image and the corresponding three-dimensional point cloud data in three-dimensional reconstruction are correctly matched.

[0034] In the real-time pose tracking link, ORB-SLAM3 algorithm is used to construct the topological relationship between local map and global map: the motion correlation between key frames is established through ORB feature extraction and matching, and the robustness of pose estimation in dynamic scenes is improved by combining IMU pre-integral constraint. A Kalman filter is introduced to predict the camera motion trajectory (prediction duration 300ms), and an ICP point cloud registration algorithm is used for real-time correction, so that the cumulative error of pose estimation is controlled within 3cm (80% improvement over traditional SLAM scheme).

[0035] By optimizing the camera internal and external parameters, the alignment accuracy of the image is further improved. On the basis of MVS and SLAM algorithms, the camera internal and external parameters are optimized to eliminate errors and improve the accuracy of three-dimensional point cloud. This step ensures that even if there is a small error in the image acquisition process, it will not affect the final three-dimensional reconstruction effect.

[0036] In combination with the fusion and supplement of depth information, the depth data collected by the laser radar sensor fills in the three-dimensional information of the areas with insufficient illumination or missing texture inside the pipeline. The depth compensation method of the radar data fuses with the depth estimation results of the panoramic image to ensure the integrity and accuracy of the three-dimensional reconstruction data in low-texture or complex environments.

[0037] An incremental three-dimensional reconstruction method is adopted to continuously add new images and depth data at each collection point, gradually improving and refining the three-dimensional reconstruction model. The incremental method ensures the generation of a more accurate three-dimensional model in the pipeline and can flexibly adjust the reconstruction accuracy according to the complexity and real-time changes of the environment inside the pipeline.

[0038] S5, performing incremental Bundle Adjustment algorithm optimization on the generated point cloud data, and using an improved Poisson Surface Reconstruction algorithm to reconstruct a smooth and continuous three-dimensional surface; The incremental Bundle Adjustment algorithm is used to globally optimize the three-dimensional point cloud and camera pose, reducing the reprojection error. The incremental BA algorithm continuously optimizes the matching accuracy of the camera pose and the three-dimensional point cloud by minimizing the geometric error between each feature point in the image and the camera, ensuring the maximum consistency of all collected images and point cloud data inside the pipeline, and reducing the error accumulation caused by noise or measurement error.

[0039] The spatial position of the entire point cloud model is optimized through graph optimization techniques such as Pose Graph Optimization, adjusting the spatial constraints of the point cloud and the camera to ensure the global consistency of the three-dimensional reconstruction. Graph optimization techniques can establish a graph model by combining different images and measurement results, optimizing the relationship between all images and point clouds, ensuring accurate matching of various areas inside the pipeline, and avoiding the expansion of local errors affecting the accuracy of the overall model.

[0040] The Poisson Surface Reconstruction algorithm is used for three-dimensional surface reconstruction, which can generate smooth and continuous three-dimensional surfaces, especially suitable for complex and irregular shaped areas inside the pipeline. Poisson Surface Reconstruction calculates surface normals from sparse point clouds to generate smooth surfaces, solving the problem of traditional surface reconstruction methods that are difficult to handle noisy data and irregular point clouds.

[0041] In combination with the multi-resolution technique, the surface reconstruction effect is optimized. For the details of complex areas, the reconstruction is refined in a multi-resolution manner to ensure that the details of key areas such as pipe joints and cracks can be accurately reconstructed. The multi-resolution technique can dynamically adjust the reconstruction accuracy according to the density and complexity of the point cloud, balancing the calculation efficiency and the reconstruction effect.

[0042] Through a depth compensation algorithm, the depth data of low-texture, unevenly illuminated, or occluded areas is supplemented to ensure that reliable three-dimensional data can be obtained in these areas. The compensation algorithm can combine lidar data and image depth information to automatically fill in missing depth information in environments with insufficient light or complex structures, thereby completing the complete three-dimensional reconstruction.

[0043] S6, based on the processed three-dimensional data, using a convolutional neural network to detect cracks and corrosion defects in the pipeline, and automatically generating defect types, locations and repair suggestions; Through a convolutional neural network (CNN), cracks, corrosion, settlement and other defects in the pipeline are automatically identified, the types, locations and severity of the defects are analyzed, and repair recommendations are generated according to the detection results to assist pipeline maintenance personnel in making scientific decisions. In addition, the defect area is enhanced and displayed in combination with image and point cloud data to improve the visualization effect of the defects, facilitating efficient evaluation and repair by the operator.

[0044] Specifically, in the defect detection link, a fusion network architecture based on ResNet50+U-Net is constructed: input a preprocessed image with a resolution of 512x512, extract multi-scale features (including 16x16, 8x8, 4x4 three down-sampling scales) through the ResNet50 backbone network, introduce an attention mechanism (CBAM module) in the decoding stage to dynamically allocate feature weights, and focus on enhancing the defect response in low-texture areas. The network outputs a binary segmentation mask of cracks and corrosion, and removes noise interference through morphological opening and closing operation (kernel size 3x3) to generate the final defect region label map.

[0045] To solve the problem of missing small defects (width < 2mm), an adversarial training strategy is designed: a synthetic defect dataset containing interference factors such as metal pipe wall reflection and oil covering is constructed, the Wasserstein GAN framework is used to optimize the segmentation boundary, and the IoU in the fuzzy edge area is improved by 37%. At the same time, a transfer learning mechanism is introduced to migrate the model weights pre-trained on natural image datasets to the pipeline scene, and the last three layers of convolution kernel parameters are fine-tuned to adapt to the pipeline texture features.

[0046] In the defect quantification link, multi-modal data of point cloud and image are fused: point cloud region is extracted based on defect segmentation mask, covariance matrix of the included angle between normal vector and image gradient direction is calculated, and geometric model of defect depth and volume is established. For irregular corrosion area, Marching Cubes algorithm is used to extract isosurface, and region growing method is used to optimize surface topological structure, so that the quantification accuracy of corrosion volume error is less than 4%.

[0047] In order to meet the real-time requirement, the double strategies of model lightening and hardware acceleration are adopted: channel pruning (compression rate 60%) and quantization (INT8 precision) are carried out on the segmentation network, and the inference speed is improved to 150ms / frame; the point cloud processing task is distributed to the CUDA core of GPU, and the parallel execution of image segmentation and point cloud quantification is realized.

[0048] S7, store and manage image and three-dimensional point cloud data through cloud platform, visualize the data by combining augmented reality or virtual reality technology, and support remote access and real-time monitoring; In the data uploading cloud link, the edge computing node cleans and compresses the detection data, and then uploads the data to the cloud server through public network (such as 5G / 4G). The cloud end uses a standardized database to store and manage the pipeline detection data, and supports searching historical data according to time, location, defect type and other dimensions.

[0049] In the visualization interaction link, a general three-dimensional visualization tool is used to load the pipeline three-dimensional model, and the user can view the defect labeling information (such as location, type, severity level). The system provides basic filtering and sorting functions to help users quickly locate high-risk areas.

[0050] In the decision support link, based on the pre-set industry standards and expert experience library, the detection conclusion (such as suggestion for repair or continuous monitoring) is automatically generated, and the defect statistical results (such as defect number, distribution area, priority ranking) are output in the form of table.

[0051] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above-mentioned teaching or related art or knowledge. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the present application shall be within the scope of protection of the appended claims of the present application.

Claims

1. A pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision, characterized in that, Comprise: S1, using panoramic camera and radar sensor for pipeline panoramic image and depth data acquisition; S2, the collected panoramic image is denoised, distortion correction and light compensation processing, and optimization; S3, using SuperPoint feature extraction algorithm to extract stable feature points in panoramic image, and using SuperGlue algorithm for feature point matching; S4, after feature point matching, combined with multi-view stereo vision algorithm and SLAM technology, using panoramic image and depth data for pose estimation and three-dimensional point cloud data generation; S5, the generated three-dimensional point cloud data is optimized by incremental Bundle Adjustment algorithm, and the improved Poisson Surface Reconstruction algorithm is used to reconstruct smooth and continuous three-dimensional surface; S6, based on the three-dimensional point cloud data processed in step S5, using convolutional neural network for intelligent detection of cracks and corrosion defects in the pipeline, and automatically generating defect type, location and repair suggestion; S7, through cloud platform storage and management of image and three-dimensional point cloud data, combined with augmented reality or virtual reality technology for visual display of data, and support remote access and real-time monitoring.

2. The pipeline 3D reconstruction and intelligent detection method based on panoramic stereo vision according to claim 1, characterized in that, Step S1 specifically includes: Using panoramic camera and radar sensor, synchronous acquisition of pipeline panoramic image and depth data, panoramic camera equipped with 8 groups of 1 / 2.3 inch CMOS sensor, supporting 5760x2880 ultra-high definition resolution, equipped with F2.8 large aperture fisheye lens, built-in gyroscope and accelerometer to assist pose solution, through dynamic exposure adjustment to adapt to 0.1 lux-100klux wide dynamic light environment, radar sensor with 360° horizontal scanning range and 0.1° vertical resolution, through GPIO interface and panoramic camera to realize nanosecond level time synchronization, equipped with multi-echo detection technology to distinguish liquid, solid sediment and metal pipe wall reflection signal, and equipped with waterproof aviation connector to adapt to wet environment, radar components are installed at the end of mechanical arm, and the camera is at a 90° angle to form a complementary observation angle. 3.The pipeline 3D reconstruction and intelligent detection method based on panoramic stereo vision according to claim 1, characterized in that, Step S2 specifically includes: Using DnCNN deep learning model to denoise the collected panoramic image, DnCNN deep learning model uses convolutional neural network structure to remove Gaussian noise and salt and pepper noise in panoramic image; Through distortion correction algorithm, the collected panoramic image is geometrically corrected, the radial and tangential distortion caused by fisheye lens is removed, and the true geometric shape of the panoramic image is restored; Adaptive light compensation algorithm is adopted to optimize the brightness and contrast of the image, in low light intensity, the exposure and brightness are automatically adjusted, the visibility of the image is enhanced, and the local brightness enhancement technology is combined to enhance the local brightness of the pipeline setting area; Combined with image cropping technology, the panoramic image is cropped, focusing on the setting part of the pipeline, after cropping, image enhancement algorithm is used to improve the contrast and clarity of the panoramic image, and highlight the setting part information in the panoramic image.

4. The pipeline 3D reconstruction and intelligent detection method based on panoramic stereo vision according to claim 1, characterized in that, Step S3 specifically includes: SuperPoint feature extraction algorithm is used to automatically extract stable feature points from the panoramic images of the pipeline; SuperGlue algorithm is used to match the feature points extracted under different perspectives, ensuring accurate alignment between images; Random Sample Consensus algorithm is used to remove matching points that do not meet geometric constraints, and then Bundle Adjustment algorithm is used to optimize the matched feature points and camera pose.

5. The method of claim 1, wherein, Step S4 specifically includes: Multi-view stereo vision algorithm is used to estimate the depth of panoramic images from different perspectives, and high-density three-dimensional point cloud data is generated through multi-perspective image alignment and reconstruction; SLAM technology is combined to estimate the pose of panoramic image data, and the combination of image and depth information is optimized through SLAM technology to realize accurate calculation of the motion trajectory of the panoramic camera in the pipeline. In the real-time pose tracking link, ORB-SLAM3 algorithm is used to construct the topological relationship between local and global maps, and the motion correlation between key frames is established through ORB feature extraction and matching. Combined with IMU pre-integration constraint, the robustness of pose estimation in dynamic scenes is improved. Kalman filter is introduced to predict the camera motion trajectory, and ICP point cloud registration algorithm is used for real-time correction; Based on MVS and SLAM algorithms, the camera internal and external parameters are optimized to eliminate errors and improve the accuracy of three-dimensional point cloud; Depth data collected by radar sensors is used to fill in the three-dimensional information of areas with insufficient lighting or missing texture in the pipeline; Incremental three-dimensional reconstruction method is used to continuously add new images and depth data at each collection point, gradually completing three-dimensional reconstruction. 6.The pipeline 3D reconstruction and intelligent detection method based on panoramic stereo vision according to claim 1, characterized in that, Step S5 specifically includes: Incremental Bundle Adjustment algorithm is used to globally optimize three-dimensional point cloud and camera pose, and the matching accuracy of camera pose and three-dimensional point cloud is continuously optimized by minimizing the geometric error between each feature point in panoramic image and camera, ensuring the maximum consistency of all collected panoramic images and point cloud data in the pipeline; Graph optimization technology is used to optimize the spatial position of the entire point cloud model, adjust the spatial constraints of point cloud and panoramic camera, and ensure the global consistency of three-dimensional reconstruction; Poisson Surface Reconstruction algorithm is used to calculate surface normals from sparse point cloud, and then generate smooth surfaces; Multi-resolution technology is combined to optimize surface reconstruction effect, and the details of complex areas are refined through multi-resolution reconstruction; Depth compensation algorithm is used to supplement the depth data of low-texture, unevenly lit or occluded areas.

Images