A pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision

Abstract

The present application relates to the field of pipeline three-dimensional reconstruction and intelligent detection, and particularly relates to a pipeline three-dimensional reconstruction and intelligent detection method based on panoramic stereo vision. The technical scheme comprises: using a panoramic camera and a radar sensor to collect panoramic images and depth data inside a pipeline; performing denoising, distortion correction and illumination compensation processing on the collected panoramic images, and performing image optimization; extracting stable feature points in the panoramic images and performing feature point matching; combining a multi-view stereo vision algorithm and a simultaneous localization and mapping technology, and using the panoramic images and the depth data to perform pose estimation and three-dimensional point cloud generation; optimizing the generated point cloud data, and reconstructing a smooth and continuous three-dimensional surface; based on the processed three-dimensional data, using a convolutional neural network to perform intelligent detection of cracks and corrosion defects inside the pipeline, and automatically generating defect types, positions and repair suggestions. The present application is suitable for pipeline detection.

Description

Technical Field

[0001] This invention relates to the field of pipeline 3D reconstruction and intelligent inspection, specifically to a pipeline 3D reconstruction and intelligent inspection method based on panoramic stereo vision. Background Technology

[0002] From a technical perspective, existing pipeline inspection technologies have significant shortcomings in multimodal sensor fusion, adaptability to complex environments, and intelligent defect analysis. Regarding multi-sensor data synchronization, current solutions mostly employ discrete sensor configurations, lacking efficient spatiotemporal calibration mechanisms. This leads to accumulated errors in image and depth information registration, making it difficult to meet high-precision reconstruction requirements, especially in scenarios involving dynamic pipeline deformation. Feature matching in low-texture areas is another technical challenge. Traditional algorithms are prone to failure on smooth metal pipe walls or surfaces covered in dirt, resulting in lost keyframes or a surge in mismatch rates during 3D reconstruction. Real-time challenges in dynamic environments stem from the computational complexity of SLAM (Simultaneous Localization and Mapping) algorithms in complex topologies. When pipelines have numerous branches or obstacles, the system often falls into local convergence or tracking loss states. Furthermore, most existing defect detection models are based on shallow feature extraction, making them sensitive to changes in lighting and noise interference, and lacking a deep understanding of the development patterns of pipeline defects, resulting in high false negative and false positive rates. These technological bottlenecks make it difficult for existing systems to meet the urgent needs of modern urban pipeline network management, and breakthroughs are urgently needed through multidisciplinary integration and independent innovation.

[0003] In summary, existing detection technologies have the following drawbacks:

[0004] Pain Point 1: Failure of Multimodal Sensing Data Fusion. In existing solutions, spatiotemporal calibration errors between optical cameras and lidar can easily lead to data registration deviations. In dynamic deformation scenarios of pipelines, relative pose drift between sensors can cause 3D reconstruction breaks, and it is difficult to achieve cross-modal feature complementarity in low-texture regions.

[0005] Pain Point 2: Insufficient perception of low-texture environments. Traditional feature extraction algorithms cannot stably match smooth metal pipe walls or dirt-covered surfaces, resulting in keyframe loss and trajectory jumps, failing to meet the high-precision pose estimation requirements in complex scenarios such as pipe bends and corrosion.

[0006] Pain Point 3: Poor real-time processing and adaptability to complex topologies. Existing SLAM algorithms have excessively high computational complexity in complex topologies such as pipe bifurcation and narrowing, resulting in system response delays exceeding 2 seconds per frame. Furthermore, they cannot balance reconstruction accuracy and efficiency through incremental optimization, thus limiting the feasibility of application in dynamic monitoring scenarios. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision, which realizes high-precision three-dimensional modeling and intelligent defect diagnosis in the entire space of the pipeline.

[0008] The present invention achieves the above objectives by adopting the following technical solution: The present invention provides a method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision, comprising:

[0009] S1. Use panoramic cameras and radar sensors to collect panoramic images and depth data inside the pipeline;

[0010] S2. Denoise, distortion correction, and illumination compensation are performed on the acquired panoramic images, and image optimization is also performed.

[0011] S3. Use the SuperPoint feature extraction algorithm to extract stable feature points in the panoramic image, and use the SuperGlue algorithm to perform feature point matching.

[0012] S4. Combining multi-view stereo vision algorithms and SLAM (Simultaneous Localization and Mapping) technology, pose estimation and 3D point cloud generation are performed using panoramic images and depth data.

[0013] S5. The generated point cloud data is optimized using an incremental Bundle Adjustment algorithm, and a smooth and continuous three-dimensional surface is reconstructed using an improved Poisson Surface Reconstruction algorithm.

[0014] S6. Based on the 3D data processed in step S5, use a convolutional neural network to intelligently detect cracks and corrosion defects in the pipeline, and automatically generate defect types, locations and repair suggestions.

[0015] S7 stores and manages image and 3D point cloud data through a cloud platform, combines augmented reality or virtual reality technology to visualize the data, and supports remote access and real-time monitoring.

[0016] Furthermore, step S1 specifically includes:

[0017] A panoramic camera and radar sensors are used to simultaneously acquire panoramic images and depth data inside the pipeline. The panoramic camera is equipped with eight 1 / 2.3-inch CMOS sensors, supports an ultra-high-definition resolution of 5760×2880, features a large-aperture fisheye lens (F2.8), and incorporates a gyroscope and accelerometer to assist in pose calculation. It adapts to a wide dynamic range of lighting environments (0.1 lux to 100 klux) through dynamic exposure adjustment. The radar sensor has a 360° horizontal scanning range and a 0.1° vertical resolution. It achieves nanosecond-level time synchronization with the panoramic camera through a GPIO interface, incorporates multi-echo detection technology to distinguish between liquid, solid deposits, and metal pipe wall reflections, and is equipped with a waterproof aviation connector to adapt to humid environments. A radar component is installed at the end of the robotic arm, forming a complementary observation perspective with the camera at a 90° angle.

[0018] Furthermore, step S2 specifically includes:

[0019] The DnCNN deep learning model is used to denoise the acquired panoramic images. The DnCNN deep learning model adopts a convolutional neural network structure to remove Gaussian noise and salt-and-pepper noise from the panoramic images.

[0020] The acquired panoramic image is geometrically corrected using a distortion correction algorithm to remove radial and tangential distortions caused by the fisheye lens and restore the true geometric shape of the panoramic image.

[0021] An adaptive illumination compensation algorithm is used to optimize the brightness and contrast of the image. Under low illumination, the exposure and brightness are automatically adjusted to enhance the visibility of the image. In addition, local enhancement technology is used to enhance the local brightness of the set area inside the pipe.

[0022] By combining image cropping technology, the panoramic image is cropped to focus on the pipe settings. After cropping, image enhancement algorithms are used to improve the contrast and clarity of the panoramic image, highlighting the settings information in the panoramic image.

[0023] Furthermore, step S3 specifically includes:

[0024] The SuperPoint feature extraction algorithm is used to automatically extract stable feature points from panoramic images of pipelines.

[0025] The SuperGlue algorithm is used to match feature points extracted from different viewpoints to ensure accurate alignment between images;

[0026] The random sampling consensus algorithm removes matching points that do not conform to geometric constraints, and then the BundleAdjustment algorithm optimizes the matched feature points and camera pose.

[0027] Furthermore, step S4 specifically includes:

[0028] Multi-view stereo vision algorithms are used to estimate the depth of panoramic images from different perspectives. High-density 3D point cloud data is generated by pixel alignment and reconstruction of multi-view images.

[0029] Combining SLAM technology, pose estimation is performed on panoramic image data. By optimizing the combination of image and depth information through SLAM technology, the motion trajectory of the panoramic camera within the pipeline can be accurately calculated. In the real-time pose tracking stage, the ORB-SLAM3 algorithm is used to construct the topological relationship between the local map and the global map. The motion association between key frames is established through ORB (Oriented FAST and Rotated BRIEF) feature extraction and matching. The robustness of pose estimation in dynamic scenes is improved by combining IMU (IMU Pre-integration Constraint) pre-integration constraints. A Kalman filter is introduced to predict the camera motion trajectory, and the ICP point cloud registration algorithm is combined for real-time correction.

[0030] Based on the MVS and SLAM algorithms, errors are eliminated and the accuracy of 3D point clouds is improved by optimizing the camera's intrinsic and extrinsic parameters.

[0031] Depth data collected by radar sensors is used to fill in the 3D information of areas inside the pipe that are poorly lit or lack texture.

[0032] An incremental 3D reconstruction method is adopted, in which new images and depth data are continuously added at each acquisition point to gradually complete the 3D reconstruction.

[0033] Furthermore, step S5 specifically includes:

[0034] The incremental Bundle Adjustment algorithm is used to globally optimize the 3D point cloud and camera pose. By minimizing the geometric error between each feature point in the panoramic image and the camera, the incremental Bundle Adjustment algorithm continuously optimizes the matching accuracy between the camera pose and the 3D point cloud, ensuring the maximum consistency of all panoramic images and point cloud data acquired in the pipeline.

[0035] The spatial position of the entire point cloud model is optimized by graph optimization technology, and the spatial constraints between the point cloud and the panoramic camera are adjusted to ensure the global consistency of the 3D reconstruction.

[0036] The Poisson Surface Reconstruction algorithm is used to calculate surface normals from sparse point clouds, thereby generating smooth surfaces;

[0037] By combining multi-resolution technology to optimize surface reconstruction results, the detailed parts of complex areas are refined through multi-resolution reconstruction.

[0038] Depth compensation algorithms are used to supplement depth data in areas with low texture, uneven lighting, or occlusion.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention employs a spatiotemporal synchronous acquisition mechanism of a panoramic camera and a LiDAR, and solves the problem of feature matching failure in low-texture areas through a dynamic error compensation algorithm. Even in scenarios with pipe bends or dirt coverage, it can still stably acquire 3D data. It introduces Retinex photometric distortion correction and adaptive exposure control technology to achieve stable feature extraction under extremely low light conditions of 0.1 lux. Combined with feature matching algorithms, it effectively improves the image registration accuracy in complex scenes.

[0041] This invention integrates MVS-SLAM3 and ICP point cloud registration technologies, and introduces Kalman filtering to predict camera motion states, significantly improving the reconstruction stability of complex topological structures. It constructs a multi-scale deep learning model to achieve pixel-level segmentation and 3D quantitative analysis of pipeline defects, providing a highly reliable basis for defect detection. Finally, through an augmented reality visualization platform and intelligent evaluation model, it enables remote operation and maintenance management and scientific decision-making, improving pipeline maintenance efficiency. At the hardware level, the system integrates a panoramic camera, LiDAR, and a robotic arm. The data processing module relies on a high-performance computing platform to complete real-time feature extraction and 3D reconstruction, while cloud management utilizes the Internet of Things and a visualization engine to achieve data storage, analysis, and decision support. This invention breaks through the environmental adaptability bottleneck of traditional detection equipment, providing an efficient and reliable intelligent solution for the refined operation and maintenance of urban infrastructure. Attached Figure Description

[0042] Figure 1 This is a flowchart of a pipeline 3D reconstruction and intelligent detection method based on panoramic stereo vision provided by an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0044] This invention provides a method for 3D reconstruction and intelligent detection of pipelines based on panoramic stereo vision, such as... Figure 1 As shown, it specifically includes:

[0045] S1. Use panoramic cameras and radar sensors to collect panoramic images and depth data inside the pipeline;

[0046] Specifically, this invention uses a panoramic camera and radar sensors to simultaneously acquire panoramic images and depth data within the pipeline. The panoramic camera is equipped with eight 1 / 2.3-inch CMOS sensors, supporting an ultra-high-definition resolution of 5760×2880 (equivalent to 28 million pixels in a monocular camera). It features an F2.8 large-aperture fisheye lens (180° horizontal / vertical field of view) and incorporates a gyroscope (±0.5° / s accuracy) and an accelerometer (±2g accuracy) to assist in pose calculation. Dynamic exposure adjustment (ISO 100-3200 adjustable, shutter speed 1 / 30s-1 / 1000s) adapts to a wide dynamic range of lighting conditions from 0.1 lux to 100 klux, and the FlowState image stabilization algorithm enables seamless stitching at pipeline bends. The device is directly connected to the data processing module via Gigabit Ethernet, with data transmission latency controlled within 50ms, and supports 5G / LoRa dual-mode wireless transmission (with resume capability).

[0047] The radar sensor features a 360° horizontal scanning range and a 0.1° vertical resolution, a ranging accuracy of ±2mm@10m, a scanning frequency of 20Hz, and a point cloud density supporting 120-1200pts / m². 2 Adjustable. This radar achieves nanosecond-level time synchronization with a panoramic camera via a GPIO interface, incorporates multi-echo detection technology to distinguish between liquids (sewage), solid sediments (sludge), and metal pipe wall reflections, and is equipped with a waterproof aviation connector (IP68 protection rating) to adapt to humid environments. The radar component is mounted at the end of the robotic arm, forming a complementary observation angle with the camera at a 90° angle, and the scanning frequency is increased to 30Hz in dynamic obstacle avoidance mode.

[0048] To achieve spatiotemporal registration of multimodal data, this invention employs a checkerboard calibration board (including infrared reflection markers) to complete camera-radar extrinsic parameter calibration (translation error ≤ 0.3 mm, rotation error ≤ 0.05°), and develops a deep learning-based distortion correction model (U-Net architecture) to optimize radial distortion coefficients k1 and k2 to within 0.001. Through the ROS middleware publish / subscribe message mechanism, combined with a GPS timing module (±1 μs accuracy), timestamp synchronization between image and point cloud data is achieved, ensuring strict alignment of their frame numbers. This synchronization mechanism supports multi-sensor data fusion at pipeline bifurcation points, effectively avoiding SLAM trajectory splitting issues.

[0049] S2. Denoise, distortion correction, and illumination compensation are performed on the acquired panoramic images, and image optimization is also performed.

[0050] The DnCNN deep learning model is used to denoise the acquired panoramic images. Noise features are extracted through convolutional layers. The training dataset contains low-light pipe images, which effectively suppresses Gaussian noise and salt-and-pepper noise.

[0051] Based on the Retinex photometric distortion correction theory, combined with adaptive filtering to separate the incident and reflected light components, the distortion correction algorithm is used to perform geometric correction on the acquired panoramic image, remove the radial and tangential distortions caused by the fisheye lens, and restore the true geometric shape of the panoramic image.

[0052] The specific steps are as follows:

[0053] Retinex light field decomposition is used to receive denoised images in order to increase details in dark areas of the image;

[0054] Geometric distortion correction involves applying calibration parameters to correct fisheye distortion, while simultaneously compensating for special pipe sections to reduce distortion.

[0055] An adaptive illumination compensation algorithm is used to optimize the brightness and contrast of the image. Under low illumination, the exposure and brightness are automatically adjusted to enhance the visibility of the image. In addition, local enhancement technology is used to enhance the local brightness of the set area inside the pipe.

[0056] By combining image cropping technology, the panoramic image is cropped to focus on the pipe settings. After cropping, image enhancement algorithms are used to improve the contrast and clarity of the panoramic image, highlighting the settings information in the panoramic image.

[0057] Spatial downsampling technology is used to compress point cloud data volume, and the voxel size is set to a fixed resolution, which significantly reduces the subsequent computational complexity;

[0058] Outliers are removed by calculating the density of neighboring points, while retaining the effective geometric features of the pipe's inner wall.

[0059] The iterative nearest point algorithm is used to optimize the mapping relationship between point cloud and image pixels, and a convergence threshold is set to realize rigid body transformation;

[0060] A pixel-point cloud projection matrix is ​​established, and depth information is mapped to the image pixel level through perspective transformation to generate an enhanced image with 3D coordinate annotation, which supports subsequent feature matching and defect localization.

[0061] S3. Use the SuperPoint feature extraction algorithm to extract stable feature points in the panoramic image, and use the SuperGlue algorithm to perform feature point matching to ensure accurate alignment and high-quality matching of images in low-light and low-texture environments.

[0062] The SuperPoint feature extraction algorithm is used to automatically extract stable feature points from pipe images. SuperPoint effectively identifies stable feature points in areas with sparse texture or insufficient lighting within the pipe. Based on a convolutional neural network architecture, this algorithm can automatically extract high-quality feature points in low-light and complex backgrounds and generate corresponding 128-dimensional descriptors, adapting to challenges such as deformation and lighting variations in different pipe environments.

[0063] The specific steps for Superpoint to extract feature points are as follows:

[0064] Image normalization preprocessing: Accept the corrected panoramic image, normalize the image, and output a standard grayscale image;

[0065] Lightweight model inference: Segment the image grid, output a 128-dimensional descriptor, generate a feature point set, and a descriptor matrix;

[0066] Feature optimization enhancement: By using non-maximum suppression, feature points with a confidence level ≥ 0.85 are retained.

[0067] The SuperGlue algorithm is used for feature point matching, which efficiently matches feature points extracted from different viewpoints to ensure accurate alignment between images. By introducing a graph neural network, the SuperGlue algorithm optimizes the correlation between feature points, enabling accurate matching of feature points even in pipeline images with occlusion, lighting variations, and lack of texture, thus reducing the possibility of mismatches.

[0068] The specific steps for Superglue feature matching are as follows:

[0069] Constructing feature associations: Receive the feature point coordinates and descriptor matrix output by the superpoint and create a bidirectional graph structure;

[0070] Attention mechanism matching: Calculate the descriptor similarity matrix and perform multi-head attention computation to generate the matching confidence matrix;

[0071] Matching and filtering: The Sinkhorn algorithm is used to solve for optimal transmission and verify consistency.

[0072] By combining the RANSAC algorithm for mismatch removal and using the random sampling consensus algorithm to remove matching points that do not conform to geometric constraints, the high quality and accuracy of the remaining feature points are ensured. This step can automatically identify and remove erroneous matching points in pipeline images caused by noise, occlusion, or other external factors, thus optimizing the feature matching results.

[0073] To enhance image features in low-light and low-texture environments, this invention improves feature matching robustness by combining deep learning with classical algorithms. Specifically, a pre-processed image with a resolution of 256×256 is received at the output layer, and the Retinex theoretical model is used to separate the emitted light component L(x,y) and the reflected light component R(x,y), which is mathematically expressed as: I(x,y)=L(x,y)⋅R(x,y)+ϵ.

[0074] Where ϵ represents the noise term. By suppressing the reflected light component R(x,y), the interference from reflections from the metal pipe wall can be effectively reduced.

[0075] An adversarial generative network (GAN) is introduced during the training phase. Its adversarial loss function is defined as follows: a feature extraction network with 8 convolutional layers is constructed, each layer configured with a 3×3 convolutional kernel, batch normalization, and LeakyReLU activation function. The last layer outputs a 128-dimensional feature descriptor, which is then processed by L2 normalization to improve cross-frame matching robustness. To address the scarcity of low-texture data for real pipelines, this invention designs a conditional generative adversarial network training framework, comprising a generator and a discriminator: the generator takes random noise and pipeline category labels as input and outputs low-texture synthetic images simulating defects such as oil stains, rust, and scratches; the discriminator must simultaneously distinguish between real and generated images and predict the pipeline category to which the image belongs. By jointly optimizing the adversarial loss, feature matching loss, and classification loss, the generated images are ensured to possess both realism and diversity.

[0076] To achieve feature extraction from complex curved surfaces (such as pipe bends and diameter reductions), this invention introduces a multi-scale feature pyramid mechanism: the downsampling branch compresses the spatial dimension step by step through convolutional layers with a stride of 2 to capture global contour features; the upsampling branch combines detailed information with transposed convolution and skip connections, and finally dynamically adjusts the fusion ratio of features at different scales through an adaptive weight allocation module—this module allocates weights based on the gradient intensity of the local image to suppress noise interference in low-texture areas.

[0077] To meet the real-time requirements of pipeline detection (single frame processing ≤120ms), this invention adopts a dual strategy of hardware acceleration and algorithm optimization: quantization and pruning of the network improve the model inference speed by 3 times; sliding window detection divides the panoramic image into 16×16 pixel local blocks and calculates the features of each window in parallel to improve throughput; at the same time, the 32-layer convolution of the original SuperPoint network is compressed to 8 layers, reducing the number of parameters by 60% while maintaining mAP≥90%.

[0078] To address the feature failure problem caused by specular reflection from metal pipe walls, this invention integrates photometric distortion compensation: a Retinex-Net network that estimates the incident light intensity map and the reflected light reflectance map in real time, dynamically adjusts the image color balance to suppress highlight clipping artifacts; uses contextual information to fill shadow areas, and combines a semantic segmentation network to identify oil-covered areas and specifically enhance texture details.

[0079] S4. Combining multi-view stereo vision algorithms and SLAM technology, pose estimation and 3D point cloud generation are performed using panoramic images and depth data.

[0080] By combining Simultaneous Localization and Mapping (SLAM) technology, pose estimation is performed on image data. SLAM optimizes the combination of image and depth information, enabling accurate calculation of the camera's trajectory within the pipe. SLAM technology can accurately estimate the camera's trajectory and positional relationship in 3D space in unknown environments, solving the problem of dynamic environmental changes that traditional methods cannot handle, and ensuring that each image in the 3D reconstruction is correctly matched with the corresponding 3D point cloud data.

[0081] In the real-time pose tracking stage, the ORB-SLAM3 algorithm is used to construct the topological relationship between the local and global maps: motion associations between keyframes are established through ORB feature extraction and matching, and the robustness of pose estimation in dynamic scenes is improved by combining IMU pre-integration constraints. A Kalman filter is introduced to predict the camera motion trajectory (prediction time 300ms), and real-time correction is performed by combining the ICP point cloud registration algorithm, controlling the cumulative pose estimation error to within 3cm (an 80% improvement over traditional SLAM schemes).

[0082] By optimizing camera intrinsic and extrinsic parameters, the alignment accuracy of the images is further improved. Building upon the MVS and SLAM algorithms, errors are eliminated and the accuracy of the 3D point cloud is enhanced through optimization of camera intrinsic and extrinsic parameters. This step ensures that even minor errors occurring during image acquisition will not affect the final 3D reconstruction result.

[0083] By combining and supplementing depth information, depth data collected by LiDAR sensors is used to fill in the 3D information of insufficiently lit or texture-deficient areas within the pipeline. The depth compensation method for LiDAR data is fused with the depth estimation results of panoramic images to ensure the integrity and accuracy of the 3D reconstruction data in low-texture or complex environments.

[0084] An incremental 3D reconstruction method is employed, continuously adding new images and depth data at each acquisition point to gradually improve and refine the 3D reconstruction model. This incremental approach ensures the gradual generation of a more accurate 3D model within the pipeline and allows for flexible adjustment of reconstruction accuracy based on the complexity and real-time changes of the pipeline environment.

[0085] S5. The generated point cloud data is optimized using an incremental Bundle Adjustment algorithm, and a smooth and continuous three-dimensional surface is reconstructed using an improved Poisson Surface Reconstruction algorithm.

[0086] An incremental Bundle Adjustment (BA) algorithm is used to globally optimize the 3D point cloud and camera pose, reducing reprojection errors. By minimizing the geometric error between each feature point in the image and the camera, the incremental BA algorithm continuously optimizes the matching accuracy between the camera pose and the 3D point cloud, ensuring maximum consistency of all acquired images and point cloud data within the pipeline and reducing error accumulation caused by noise or measurement errors.

[0087] Graph optimization techniques (such as Pose Graph Optimization) are used to optimize the spatial position of the entire point cloud model and adjust the spatial constraints between the point cloud and the camera to ensure global consistency in 3D reconstruction. Graph optimization techniques can establish a graph model from different images and measurement results, optimize the relationship between all images and point clouds, ensure accurate matching of various regions within the pipeline, and avoid the spread of local errors from affecting the accuracy of the overall model.

[0088] The Poisson Surface Reconstruction algorithm is used for 3D surface reconstruction. This algorithm can generate smooth and continuous 3D surfaces, and is particularly suitable for complex and irregularly shaped areas inside pipes. Poisson Surface Reconstruction generates smooth surfaces by calculating surface normals from sparse point clouds, thus overcoming the shortcomings of traditional surface reconstruction methods in handling noisy data and irregular point clouds.

[0089] By combining multi-resolution technology to optimize surface reconstruction results, detailed parts of complex areas are refined through multi-resolution reconstruction, ensuring that details in critical areas (such as pipe joints and cracks) can be accurately reconstructed. Multi-resolution technology can dynamically adjust reconstruction accuracy based on the density and complexity of the point cloud, balancing computational efficiency and reconstruction results.

[0090] A depth compensation algorithm is used to supplement depth data in areas with low texture, uneven lighting, or occlusion, ensuring reliable 3D data can be obtained even in these areas. This compensation algorithm can combine LiDAR data with image depth information to automatically fill in missing depth information in environments with insufficient light or complex structures, thereby completing a full 3D reconstruction.

[0091] S6. Based on the processed 3D data, a convolutional neural network is used to intelligently detect cracks and corrosion defects in the pipeline, and automatically generate defect types, locations and repair suggestions.

[0092] Convolutional neural networks (CNNs) are used to automatically identify defects such as cracks, corrosion, and settlement within pipelines. The system analyzes the type, location, and severity of these defects and generates repair recommendations based on the detection results, assisting pipeline maintenance personnel in making informed decisions. Furthermore, image and point cloud data are combined to enhance the display of defect areas, improving the visualization of defects and facilitating efficient assessment and repair by operators.

[0093] Specifically, in the defect detection stage, a fusion network architecture based on ResNet50+U-Net is constructed: A preprocessed image with a resolution of 512×512 is input, and multi-scale features (including three downsampling scales: 16×16, 8×8, and 4×4) are extracted through the ResNet50 backbone network. During the decoding stage, an attention mechanism (CBAM module) is introduced to dynamically allocate feature weights, focusing on enhancing the defect response in low-texture regions. The network outputs a binarized segmentation mask for cracks and erosion, and noise interference is removed through morphological opening and closing operations (kernel size 3×3) to generate the final defect region annotation map.

[0094] To address the issue of missed detection of minute defects (width < 2mm), an adversarial training strategy was designed: a synthetic defect dataset containing interfering factors such as metal pipe wall reflection and oil stains was constructed, and the Wasserstein GAN framework was used to optimize the segmentation boundary, improving the IoU of the model in blurred edge regions by 37%. Simultaneously, a transfer learning mechanism was introduced, transferring the model weights pre-trained on a natural image dataset to the pipeline scene, and adapting the pipeline texture features by fine-tuning the parameters of the last three convolutional kernel layers.

[0095] In the defect quantization stage, multimodal data from point clouds and images are integrated: point cloud regions are extracted based on defect segmentation masks, and the covariance matrix of the angle between normal vectors and the image gradient direction is calculated to establish a geometric model of defect depth and volume. For irregular corrosion regions, the Marching Cubes algorithm is used to extract isosurfaces, and the surface topology is optimized by combining it with the region growing method to achieve a quantization accuracy of corrosion volume error <4%.

[0096] To meet real-time requirements, a dual strategy of model lightweighting and hardware acceleration is adopted: channel pruning (60% compression rate) and quantization (INT8 precision) are performed on the segmentation network, increasing the inference speed to 150ms / frame; point cloud processing tasks are allocated to the CUDA core of the GPU to achieve parallel execution of image segmentation and point cloud quantization.

[0097] S7. Stores and manages image and 3D point cloud data through a cloud platform, combines augmented reality or virtual reality technology to visualize the data, and supports remote access and real-time monitoring.

[0098] In the data upload process, edge computing nodes clean and compress the inspection data before uploading it to the cloud server via a public network (such as 5G / 4G). The cloud uses a standardized database to store and manage the pipeline inspection data, supporting the retrieval of historical data by time, location, defect type, and other dimensions.

[0099] In the visualization and interaction phase, a 3D model of the pipeline is loaded using a general-purpose 3D visualization tool, allowing users to view defect annotation information (such as location, type, and severity level). The system provides basic filtering and sorting functions to facilitate users in quickly locating high-risk areas.

[0100] In the decision support phase, based on pre-set industry standards and expert experience databases, detection conclusions (such as recommendations for repair or continuous monitoring) are automatically generated, and defect statistics (such as number of defects, distribution area, and priority ranking) are output in tabular form.

[0101] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for 3D reconstruction and intelligent detection of pipelines based on panoramic stereo vision, characterized in that, include: S1. Use panoramic cameras and radar sensors to collect panoramic images and depth data inside the pipeline; S2. Perform noise reduction, distortion correction, and illumination compensation on the acquired panoramic images, and optimize them accordingly. S3. Use the SuperPoint feature extraction algorithm to extract stable feature points in the panoramic image, and use the SuperGlue algorithm to perform feature point matching. S4. After feature point matching, combine multi-view stereo vision algorithm and SLAM technology to perform pose estimation and 3D point cloud data generation using panoramic images and depth data. S5. Optimize the generated 3D point cloud data using the incremental Bundle Adjustment algorithm, and reconstruct a smooth and continuous 3D surface using the improved Poisson Surface Reconstruction algorithm. S6. Based on the 3D point cloud data processed in step S5, use a convolutional neural network to intelligently detect cracks and corrosion defects in the pipeline, and automatically generate defect types, locations and repair suggestions. S7 stores and manages image and 3D point cloud data through a cloud platform, combines augmented reality or virtual reality technology to visualize the data, and supports remote access and real-time monitoring.

2. The method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision according to claim 1, characterized in that, Step S1 specifically includes: A panoramic camera and radar sensors are used to simultaneously acquire panoramic images and depth data inside the pipeline. The panoramic camera is equipped with eight 1 / 2.3-inch CMOS sensors, supports an ultra-high-definition resolution of 5760×2880, features a large-aperture fisheye lens (F2.8), and incorporates a gyroscope and accelerometer to assist in pose calculation. It adapts to a wide dynamic range of lighting environments (0.1 lux to 100 klux) through dynamic exposure adjustment. The radar sensor has a 360° horizontal scanning range and a 0.1° vertical resolution. It achieves nanosecond-level time synchronization with the panoramic camera through a GPIO interface, incorporates multi-echo detection technology to distinguish between liquid, solid deposits, and metal pipe wall reflections, and is equipped with a waterproof aviation connector to adapt to humid environments. A radar component is installed at the end of the robotic arm, forming a complementary observation perspective with the camera at a 90° angle.

3. The method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision according to claim 1, characterized in that, Step S2 specifically includes: The DnCNN deep learning model is used to denoise the acquired panoramic images. The DnCNN deep learning model adopts a convolutional neural network structure to remove Gaussian noise and salt-and-pepper noise from the panoramic images. The acquired panoramic image is geometrically corrected using a distortion correction algorithm to remove radial and tangential distortions caused by the fisheye lens and restore the true geometric shape of the panoramic image. An adaptive illumination compensation algorithm is used to optimize the brightness and contrast of the image. Under low illumination, the exposure and brightness are automatically adjusted to enhance the visibility of the image. In addition, local enhancement technology is used to enhance the local brightness of the set area inside the pipe. By combining image cropping technology, the panoramic image is cropped to focus on the pipe settings. After cropping, image enhancement algorithms are used to improve the contrast and clarity of the panoramic image, highlighting the settings information in the panoramic image.

4. The method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision according to claim 1, characterized in that, Step S3 specifically includes: The SuperPoint feature extraction algorithm is used to automatically extract stable feature points from panoramic images of pipelines. The SuperGlue algorithm is used to match feature points extracted from different viewpoints to ensure accurate alignment between images; The random sampling consensus algorithm removes matching points that do not conform to geometric constraints, and then the BundleAdjustment algorithm optimizes the matched feature points and camera pose.

5. The method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision according to claim 1, characterized in that, Step S4 specifically includes: Multi-view stereo vision algorithms are used to estimate the depth of panoramic images from different perspectives. High-density 3D point cloud data is generated by pixel alignment and reconstruction of multi-view images. Combining SLAM technology, pose estimation is performed on panoramic image data. By optimizing the combination of image and depth information through SLAM technology, the motion trajectory of the panoramic camera in the pipeline can be accurately calculated. In the real-time pose tracking stage, the ORB-SLAM3 algorithm is used to construct the topological relationship between the local map and the global map. The motion association between key frames is established through ORB feature extraction and matching. The robustness of pose estimation in dynamic scenes is improved by combining IMU pre-integration constraints. Kalman filter is introduced to predict the camera motion trajectory. Real-time correction is performed by combining ICP point cloud registration algorithm. Based on the MVS and SLAM algorithms, errors are eliminated and the accuracy of 3D point clouds is improved by optimizing the camera's intrinsic and extrinsic parameters. Depth data collected by radar sensors is used to fill in the 3D information of areas inside the pipe that are poorly lit or lack texture. An incremental 3D reconstruction method is adopted, in which new images and depth data are continuously added at each acquisition point to gradually complete the 3D reconstruction.

6. The method for three-dimensional reconstruction and intelligent detection of pipelines based on panoramic stereo vision according to claim 1, characterized in that, Step S5 specifically includes: The incremental Bundle Adjustment algorithm is used to globally optimize the 3D point cloud and camera pose. By minimizing the geometric error between each feature point in the panoramic image and the camera, the matching accuracy between the camera pose and the 3D point cloud is continuously optimized, ensuring the maximum consistency of all panoramic images and point cloud data acquired in the pipeline. The spatial position of the entire point cloud model is optimized by graph optimization technology, and the spatial constraints between the point cloud and the panoramic camera are adjusted to ensure the global consistency of the 3D reconstruction. The Poisson Surface Reconstruction algorithm is used to calculate surface normals from sparse point clouds, thereby generating smooth surfaces; By combining multi-resolution technology to optimize surface reconstruction results, the detailed parts of complex areas are refined through multi-resolution reconstruction. Depth compensation algorithms are used to supplement depth data in areas with low texture, uneven lighting, or occlusion.

Images