Methods and Devices for 3D Reconstruction and Disease Measurement of Underground Space Based on Panoramic Cameras

Abstract

This invention discloses a method and device for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera, belonging to the field of underground engineering detection and three-dimensional reconstruction technology. This invention integrates absolute scale constraints into the reconstruction process by introducing a scale calibration plate, and accurately maps the two-dimensional disease identification results to the three-dimensional model through ray intersection, solving the problems of existing technologies where the three-dimensional model lacks a true physical scale and diseases cannot be quantitatively measured. This invention restores the true physical scale through the calibration plate, achieves accurate location and labeling of diseases through ray intersection with a Gaussian ellipsoid, and finally outputs a three-dimensional model with disease labels and a structured measurement report, realizing quantitative analysis of diseases and full-process digital output.

Description

Technical Field

[0001] This invention relates to the field of underground engineering inspection and three-dimensional reconstruction technology, specifically to a method and device for three-dimensional reconstruction and defect measurement of underground space based on a panoramic camera. Background Technology

[0002] Underground spaces, water diversion tunnels, and artificial caverns are enclosed spaces that are crucial components of infrastructure, and their structural safety directly impacts operational safety. The presence of surface defects threatens structural safety and operational lifespan. Common types of defects include cracks, leaks, weathering and spalling, salt corrosion, sediment cover, and biological erosion. Cracks, in particular, not only reduce the overall load-bearing capacity of the structure but can also accelerate leaks and cause localized detachment. Prolonged leaks can lead to corrosion of the internal steel reinforcement and deterioration of the concrete lining, especially in frigid regions where repeated freeze-thaw cycles can cause lining deformation and localized damage. Therefore, regular monitoring and quantitative analysis of surface defects in tunnel walls are critical means to ensure their long-term stable operation.

[0003] Currently, the industry mainly offers the following technical solutions for the detection and 3D archiving of surface defects in enclosed spaces such as underground spaces and tunnels:

[0004] (1) Manual inspection

[0005] Inspectors carried crack width cards, measuring tapes, and other tools into the hole to conduct close-range manual measurements. While this method can obtain direct dimensional data, it suffers from problems such as low efficiency, limited coverage, and significant susceptibility to subjective factors. Furthermore, it is difficult to create a unified three-dimensional spatial archive, and the data cannot be effectively digitally managed or compared with historical records.

[0006] (2) Three-dimensional laser scanning or moving measurement system

[0007] These systems acquire 3D point cloud or image data with absolute scale using lidar or visual sensors, enabling rapid mobile detection and full-section acquisition. However, these systems are expensive, have strict requirements for operating space and visibility, and data quality is easily affected by the presence of water mist, highly reflective surfaces, or dust within the tunnel. Furthermore, the systems are bulky and have limited mobility, making them poorly adaptable to confined spaces or underground environments with obstacles, hindering their low-cost deployment in routine inspections.

[0008] (3) Photogrammetry using ordinary cameras or drones

[0009] This method generates 3D models by taking photos from multiple angles and utilizing photogrammetry techniques such as structure-of-motion reconstructive photography. The equipment is lightweight and low-cost. However, it has the following limitations: First, due to the scale ambiguity of monocular vision, the generated 3D model only has a relative scale and lacks physical units of size corresponding to the real world. To obtain a measurable metric model, a large number of control points of known size need to be set up in the scene, which is a complex process with high precision requirements. Second, to meet the image overlap requirements for reconstruction, a large number of redundant photos of the same area are needed, making the data acquisition process time-consuming and labor-intensive, and prone to reconstruction failure under conditions of missing texture or changing lighting.

[0010] (4) Disease detection methods based on video or panoramic images

[0011] 360° panoramic cameras, which cover the entire field of view, are easy to operate, and have low cost, are gradually being introduced into underground space inspections. While these methods can output location boxes or pixel-level segmentation masks for defects in two-dimensional images, the recognition results are limited to a two-dimensional plane. They cannot obtain depth information and true dimensions of defects in three-dimensional space, making it difficult to determine the actual length, width, and spatial orientation of cracks on the curved surface of the tunnel wall. Therefore, they cannot meet the needs of quantitative defect assessment in engineering applications.

[0012] In summary, existing technical solutions have shortcomings in terms of absolute scale acquisition, data collection efficiency, environmental adaptability, and three-dimensional quantitative measurement capabilities, making it difficult to achieve accurate three-dimensional quantitative measurement of apparent diseases in underground spaces at a low cost. Summary of the Invention

[0013] To address the aforementioned problems, the purpose of this invention is to provide a method and apparatus for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera. By introducing a scale calibration plate, the scale ambiguity of pure visual reconstruction is eliminated, and two-dimensional disease information is mapped to a three-dimensional model with real scale, thereby achieving accurate geometric measurement and quantitative analysis of the disease.

[0014] This was achieved through the following technical solutions:

[0015] A method for three-dimensional reconstruction and disease measurement of underground space based on panoramic cameras includes the following steps:

[0016] Step S1: Path planning and calibration board setup, outputting the shooting path and scale calibration board setup scheme, the setup scheme includes the spatial position and orientation of at least one scale calibration board with known physical dimensions;

[0017] Step S2: Panoramic image acquisition and preprocessing. Acquire a sequence of panoramic images of the underground space along the shooting path, preprocess the acquired panoramic image sequence, and output the preprocessed panoramic image sequence.

[0018] Step S3: Sparse reconstruction and camera parameter estimation. Feature extraction and matching are performed on the preprocessed panoramic image sequence. The sparse 3D point cloud and camera pose of the scene are restored using the motion recovery structure framework. The camera pose and sparse point cloud at a relative scale are obtained and output.

[0019] Step S4: Scale restoration. Using the known physical size of the scale calibration board, the feature points of the scale calibration board in the preprocessed panoramic image sequence are detected, the global scale factor is calculated, and the global scale factor is used as a constraint to jointly optimize the camera pose and sparse point cloud with relative scale, and output the camera pose and sparse point cloud with absolute physical scale.

[0020] Step S5: Gaussian splash dense reconstruction. Based on the sparse point cloud with absolute physical scale, a dense three-dimensional model of the underground space is generated using three-dimensional Gaussian splash technology.

[0021] Step S6: Two-dimensional disease identification. Disease detection and segmentation are performed on the preprocessed panoramic image sequence, and the two-dimensional disease identification results are output. The two-dimensional disease identification results include disease category, bounding box or pixel-level mask.

[0022] Step S7: Map the two-dimensional results to three dimensions. Using the camera pose with absolute physical scale, back-project the two-dimensional disease identification results onto the dense three-dimensional model. Determine the three-dimensional interval corresponding to the disease by intersecting the ray with the model and assign disease labels. Output a three-dimensional model with disease semantic labels.

[0023] Step S8: 3D disease measurement and report output. On the 3D model with disease semantic tags, perform geometric measurements on the disease and output the 3D model with disease semantic tags and a structured measurement report.

[0024] Optionally, the path planning and scale calibration board deployment includes: generating a sequence of shooting points and a travel path based on the prior physical dimensions of the underground space to be inspected, and outputting a scale calibration board deployment scheme, which includes the number, spatial location and orientation of the calibration boards, to ensure that the calibration boards are clearly imaged in panoramic images of multiple consecutive shooting points.

[0025] Optionally, panoramic image acquisition and preprocessing includes: acquiring a sequence of panoramic images of the underground space along a preset shooting path, and flattening, cropping, and correcting distortion of the acquired panoramic images; generating a noise mask using a multimodal information fusion mechanism combining rule thresholding and semantic segmentation models to automatically block interfering targets in the panoramic images, thereby effectively suppressing the interference of uneven lighting and moving objects in the cave on the reconstruction process and improving the robustness of subsequent reconstruction and recognition. Interfering targets include inspection personnel, equipment, and strong light spots.

[0026] Optionally, joint optimization is performed on the camera pose and sparse point cloud at a relative scale. Specifically, the global scale factor is incorporated as a constraint term into the bundle adjustment optimization process. By constructing and solving the joint optimization function, the camera pose, 3D point coordinates and scale factor are optimized simultaneously, and the camera pose and sparse point cloud at a relative scale are transformed into camera pose and sparse point cloud at an absolute physical scale.

[0027] Optionally, the dense 3D model uses the camera pose and sparse point cloud with absolute physical scale as the geometric reference, so that the dense 3D model maintains the metric scale and provides a geometric basis with real physical scale for subsequent disease mapping and measurement.

[0028] Optionally, the two-dimensional disease identification specifically includes: using a lightweight hybrid deep network to detect and segment diseases in a panoramic image, and outputting disease categories, bounding boxes, or pixel-level masks; inputting the detected disease regions into a multimodal large model for refined analysis to generate a text description of the disease; and fusing the geometric information output by the lightweight hybrid deep network with the refined attributes output by the multimodal large model to form a two-dimensional disease identification result.

[0029] Optionally, mapping the 2D results to 3D includes: using the camera pose with absolute physical scale, back-projecting the 2D disease identification results to a dense 3D model; determining the 3D region corresponding to the disease by intersecting the ray with the Gaussian ellipsoid; and assigning disease labels and color codes to the corresponding 3D region; introducing multi-view projection consistency constraints, and cross-validating the disease using observations from multiple perspectives to output a 3D model with disease semantic labels. Specifically, back-projecting the 2D identification results to the dense 3D model includes: using the camera pose, emitting a ray from the camera's optical center passing through the pixels of the 2D disease region, and performing intersection calculations with multiple 3D units constituting the dense 3D model to determine the target 3D unit associated with the disease region; recoloring and labeling the target 3D unit to generate a 3D visualization model with disease semantic information, thereby establishing a linkage mechanism between the 2D disease identification results and the 3D reconstruction model, achieving accurate positioning and visual annotation of disease information in 3D space.

[0030] Optionally, geometric measurements of the defects include crack length calculation based on geodesic algorithms, crack width measurement integrating two-dimensional and three-dimensional information, and area calculation of planar defects based on surface correction. The output includes a three-dimensional model with defect semantic tags and a structured measurement report, thereby enabling automatic measurement of key geometric parameters of defects such as cracks, spalling, and leakage on a three-dimensional model with absolute scale, and outputting structured data that directly connects to the operation and maintenance management system.

[0031] Furthermore, this invention also provides a device for three-dimensional reconstruction and defect measurement of underground space based on a panoramic camera, used to implement the above-mentioned method, comprising: a mobile support platform; at least one panoramic camera unit fixed on the mobile support platform for acquiring panoramic image sequences of the underground space; at least one calibration plate with known physical dimensions deployed on a preset shooting path; a data processing unit for receiving image data acquired by the panoramic camera unit and outputting a three-dimensional semantic model with defect labels and defect quantification parameters; and at least one auxiliary device: a supplementary lighting assembly for supplementary lighting and an inertial measurement unit for assisting pose estimation. This device, by mounting the panoramic camera and calibration plate on the mobile support platform and combining the data processing unit to execute the above-mentioned method steps, forms a complete visual inspection solution with engineering-grade measurement capabilities.

[0032] The beneficial effects of this invention compared to the prior art are:

[0033] 1) Absolute scaling measurement

[0034] This achievement enables the absolute scaling of the 3D model of the tunnel wall, elevating the detection of surface defects from the traditional "visible and labelable" level to the "quantifiable and comparable" engineering application level. By deploying scale calibration boards with known physical dimensions at the data acquisition site, absolute scale constraints are introduced into the panoramic image 3D reconstruction process. This effectively eliminates the inherent scale ambiguity of pure visual reconstruction, ensuring that the generated 3D model possesses a metric scale consistent with the real world. This provides precise data support for defect development trend analysis, maintenance workload calculation, and scientific decision-making.

[0035] 2) Low-cost, lightweight deployment

[0036] This invention significantly reduces equipment costs and deployment complexity, while offering both high-efficiency data acquisition capabilities and adaptability to complex environments. Compared to complex and costly high-precision laser scanning equipment, this invention utilizes a lightweight combination of a 360° panoramic camera and calibration board, resulting in a substantial reduction in equipment costs and simplified deployment and operation. It can quickly adapt to complex operating conditions such as insufficient lighting, narrow spaces, and limited environmental constraints in underground spaces, making it more suitable for routine and periodic inspection work.

[0037] 3) Two-dimensional-three-dimensional linkage mechanism

[0038] A linkage mechanism between 2D image recognition results and 3D reconstruction models was established, realizing the dimensional upgrade of disease information from "image markers" to "spatial entities". Utilizing the recovered absolute-scale camera pose, the 2D disease recognition results were accurately mapped to the 3D Gaussian splash model through a ray-Gaussian ellipsoid intersection mechanism. This not only preserves the 2D image as a traceable original basis for disease detection, but also enables precise localization, stereo labeling, and batch statistics of diseases in 3D space, significantly enhancing the interpretability and completeness of the detection results.

[0039] 4) Strong environmental robustness

[0040] By generating a noise mask and scale constraint optimization through a multimodal information fusion mechanism in panoramic image preprocessing, the interference of uneven lighting inside the cave and moving objects on the reconstruction process is effectively suppressed. At the same time, the scale constraint is incorporated as a soft constraint into the bundle adjustment for joint optimization, which improves the geometric stability of 3D reconstruction and the repeatability consistency of disease measurement, and significantly reduces the scale drift problem between data collected at different times.

[0041] 5) End-to-end digital output

[0042] The outputs can be directly integrated with existing operation and maintenance management systems, achieving a closed-loop digital process for the entire detection data flow. The metric 3D model with defect labels, structured measurement reports, and visualization charts output by this invention are compatible with existing infrastructure operation and maintenance management systems or asset ledgers in terms of data format and content fields. The outputs include structured information such as defect type, 3D spatial location, geometric dimensions, confidence level, and evidence map index, enabling full-process digital management of detection data from acquisition and processing to archiving and application, providing comprehensive technical support for intelligent operation and maintenance and scientific upkeep of underground structures. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the overall process of an embodiment of the present invention.

[0044] Figure 2 This is a flowchart of panoramic image processing according to an embodiment of the present invention, illustrating the steps of image flattening, cropping, and noise mask generation;

[0045] Figure 3 This is a schematic diagram of the scale recovery principle in an embodiment of the present invention, illustrating the process of calibration plate feature detection, scale factor calculation, and scale constraint optimization.

[0046] Figure 4 The flowcharts for sparse reconstruction and Gaussian splashing dense reconstruction in embodiments of the present invention illustrate the reconstruction path from feature extraction to Gaussian ellipsoid optimization. Detailed Implementation

[0047] The technical solutions in the embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] like Figure 1 As shown, this invention provides a method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera, the specific steps of which include:

[0049] Step S1: Path planning and calibration board setup, outputting the shooting path and scale calibration board setup scheme:

[0050] A mobile platform (such as an inspection robot or a handheld data acquisition platform) equipped with a 360° panoramic camera enters the underground chamber to be inspected. Based on pre-inputted physical dimensions of the underground space to be inspected (such as design length, width, height, and turning radius, which can be roughly obtained through engineering drawings, simple laser rangefinders, or an initial rapid scan) as environmental constraints, the system automatically generates an optimal sequence of shooting points and a safe travel path plan, and outputs a layout scheme for scale calibration boards. The layout scheme includes the spatial location and orientation of at least one scale calibration board with known physical dimensions. Specifically, the layout scheme includes the number, spatial location, and orientation of the scale calibration boards to ensure clear imaging of the scale calibration boards in panoramic images from multiple consecutive shooting points.

[0051] The core objective of path planning is to ensure sufficient overlap between panoramic images acquired from adjacent shooting points (requiring an overlap area of ​​more than 60%) to meet the stability requirements of subsequent motion recovery structure 3D reconstruction, while maximizing operational efficiency by comprehensively covering the entire inspection area with the fewest possible acquisition points.

[0052] The calibration board is a planar reference board with a high-contrast, high-precision regular pattern printed on its surface, such as a black and white checkerboard or a coded mark with unique ID information (such as AprilTag or ArUco code), used to provide absolute scale constraints. Based on the panoramic camera's field of view model and the calculated precise shooting point pose, the system automatically calculates and recommends the optimal number, spatial position, and orientation of calibration boards through spatial geometric visibility analysis. This ensures that each calibration board can be clearly and completely imaged in panoramic images acquired from multiple (usually no less than 3) consecutive preset shooting points near its placement point, providing a reference benchmark with known physical dimensions for subsequent scale restoration.

[0053] Step S2: Panoramic image acquisition and preprocessing, outputting the preprocessed panoramic image sequence:

[0054] The mobile platform moves along a preset shooting path, while the panoramic camera simultaneously acquires a sequence of panoramic images of the underground space. The acquired raw panoramic images, in spherical or cylindrical format, undergo normalization processing (flattening, cropping, and distortion correction), as follows: Figure 2 As shown:

[0055] First, the panoramic image is flattened into a standard two-dimensional format. Then, it is determined whether the latitude-related distortion region is significant: if significant, image equalization or filtering preprocessing is performed to improve image quality; if not significant, this processing is skipped, and subsequent steps are performed.

[0056] Subsequently, based on the analysis of the cave's geometry or image content, irrelevant sky and ground portions are cropped out, retaining the effective area corresponding to the cave walls. The effective area refers to the portion of the cave wall surface in the panoramic image, and its upper and lower vertical boundaries are determined based on prior information about the underground space height, or automatically calculated through preliminary analysis of image content (such as detecting the horizon or major texture areas), thereby delineating the cave wall area to be retained.

[0057] Next, it is determined whether noisy targets (such as inspection personnel, equipment, strong light spots, floating objects, etc.) are detected. If detected, a multimodal information fusion mechanism combining rule thresholding and semantic segmentation models is used to generate a noise mask, and mask fusion and optimization are performed. Morphological operations (such as opening and closing operations) are used to remove small noise points and smooth the mask boundaries, thereby automatically shielding interfering targets in the panoramic image. If no targets are detected, mask generation is skipped, and the preprocessed image is directly output for subsequent reconstruction and recognition. Through the above flattening, cropping, and masking processes, the original panoramic image is transformed into a standardized, high-quality visual input that has removed major environmental interferences, thereby effectively suppressing the interference of uneven lighting inside the cave and moving objects on the reconstruction process, and improving the robustness of subsequent reconstruction and recognition.

[0058] Step S3: Sparse reconstruction and camera parameter estimation, outputting relative-scale camera pose and sparse point cloud:

[0059] Feature extraction and matching are performed on the preprocessed panoramic image sequence. To avoid insufficient feature points on low-texture walls, a learning-based feature extraction and description method (such as SuperPoint or D2-Net) is used to detect richer and more stable key points from the image. Figure 4 As shown, a motion reconstruction framework is used to reconstruct the sparse 3D point cloud and camera pose of the scene: The image pair with the most matching elements and the highest overlap is selected as the initial reconstruction pair. The relative rotation and translation of the two images are recovered, and their matching points are triangulated to generate the first batch of 3D spatial points. Then, unregistered images (i.e., images not yet involved in 3D reconstruction) with the most visible relationship to the currently reconstructed 3D points are continuously selected. The PnP algorithm is used to solve for their camera pose relative to the current 3D model, and new 3D points are triangulated and added to the model. The camera pose and 3D point coordinates are optimized using bundle adjustment to obtain the camera pose at a relative scale. With sparse point clouds Among them, the first... The camera rotation matrix of the image is denoted as The translation vector is denoted as ;No. A three-dimensional coordinate system is defined as These parameters are not related to the actual physical scale, but represent the camera pose at a relative scale. Therefore, they are distinguished using the superscript rel. , And rotation matrix It describes direction and is unrelated to scale, so no rel superscript is added.

[0060] Step S4: Scale recovery, outputting the camera pose and sparse point cloud with absolute physical scale:

[0061] Using the scale calibration board deployed in step S1, feature points and their known physical dimensions are detected in the preprocessed panoramic image sequence. A global scale factor is calculated, and this global scale factor is used as a constraint to jointly optimize the camera pose and sparse point cloud at relative scales. The reconstruction results at relative scales are then converted into a metric sparse model with absolute physical scales, outputting the camera pose and point cloud at absolute scales. The principle of scale recovery is as follows: Figure 3 As shown, the specific steps include:

[0062] Step S401 Calibration board feature detection and association: For each scale calibration board, detection is performed in each image in which it appears, and the two-dimensional pixel coordinates of its corner points in the image are obtained.

[0063] Step S402 Scale factor calculation: Calculate the camera pose using the relative scale of the image output from S3. and internal reference The set of three-dimensional coordinates of these corner points in the reconstructed coordinate system at the current relative scale is calculated by triangulation. , where subscript The index number of the corner point, , This represents the total number of detected corner points. The true physical distance between different corner points on the same calibration plate is known. For any pair of known true distances... Calibration plate corner Their Euclidean distance in the relative scale coordinate system is Meanwhile, the Euclidean distance in the relative scale coordinate system and the global scale factor... There is a proportional relationship: Therefore, a scaling factor estimate can be calculated from a single pair of points. :

[0064]

[0065] To improve robustness and fully utilize all observation information, a global scale factor estimation method based on weighted least squares is adopted. This method unifies all observed distance constraints into an optimization framework, yielding the optimal estimate of the global scale factor. :

[0066]

[0067] in: The total number of effective distance observations included in the calculation, i.e. the number of observations included in the calculation after quality screening. These observations may come from different sides of the same calibration plate or different calibration plates. and These are the calculated distance and the known actual physical distance of the p-th observation at a relative scale, respectively. It is to give the first The weight of each observation can be dynamically adjusted based on factors such as the reprojection error of the observation, the clarity of the calibration plate image, and the number of observations, in order to reduce the impact of false detections or low-quality observations.

[0068] After obtaining the global scale factor, the relative scale model is initially converted into a metric model:

[0069]

[0070] in, , , These represent the metric camera translation vector, metric sparse point cloud, and metric calibration plate corner coordinates, respectively, after preliminary conversion.

[0071] Step S403 Introducing Global Joint Optimization with Scale Constraints: The global scale factor calculated based on the scale calibration plate is used as a constraint term and incorporated into the bundle adjustment optimization process to construct and solve the joint optimization function.

[0072]

[0073] Among them: the first item It is the standard BA reprojection error formula, used to ensure the consistency between the model and image observations; Representing the The first image Two-dimensional pixel coordinates of each feature point; second term This is an absolute scale constraint term introduced in this invention, which forces the deviation between the reconstructed distance and the true distance; This is the total number of calibration plates observed; It is the first The set of all edges (corner pairs) with known true distances on the block calibration plate; , These are optimization variables, representing the first, second, and third generations respectively. The first on the calibration board and the The three-dimensional coordinates of each corner point in the world coordinate system; Indicates the first The camera translation vector corresponding to each image is used to represent the camera's position in the world coordinate system. In other words, 3D point coordinates represent the position of the j-th feature point in the scene within the world coordinate system. These coordinates are expressed through their corresponding 3D point variables. Or it can participate in optimization as an independent optimization parameter; It is the edge The known actual physical length; It is the length of the side calculated by the optimized model; It is a regularization weight parameter used to balance the importance of the reprojection error term and the scale constraint term. It can be set through cross-validation.

[0074] By solving this nonlinear least squares problem, and simultaneously optimizing the camera pose, 3D point coordinates, and scale factor, the output is a camera pose and sparse point cloud with absolute physical scale.

[0075] Step S5: Gaussian splash dense reconstruction to generate a metric 3D model;

[0076] Based on the absolute-scale sparse point cloud output from step S4, a dense 3D model of the underground space is generated using a 3D Gaussian splashing technique. For example... Figure 4 As shown, based on the metric point cloud, the center of the three-dimensional Gaussian ellipsoid set is initialized, that is, each three-dimensional point in the sparse point cloud is initialized as a Gaussian ellipsoid. The three-dimensional Gaussian ellipsoid is projected onto the two-dimensional image plane through differentiable rasterization rendering. The difference between the rendered image and the real image is calculated as the loss function. The Gaussian parameters (including position, shape, color and opacity) are optimized through backpropagation. Combined with adaptive densification and pruning strategies, a dense three-dimensional model with absolute physical scale is generated.

[0077] Step S6: Two-dimensional disease identification, output the two-dimensional disease identification results;

[0078] A lightweight hybrid deep network is employed to detect and segment diseases in panoramic images, outputting disease categories, bounding boxes, or pixel-level masks. The detected disease regions are then input into a multimodal large-scale model for refined analysis, generating textual descriptions of the diseases. The lightweight hybrid deep network uses an improved MobileNetV3 or ShuffleNetV2 as its backbone to extract features. After high-level features, a lightweight Transformer module is introduced to capture global contextual information. A detection head based on adaptive spatial feature fusion outputs disease categories and bounding boxes, while a lightweight segmentation head outputs pixel-level masks. The image regions defined by the detected bounding boxes or masks are cropped into image patches and input into the multimodal large-scale model for refined analysis. Structured prompts guide the multimodal large-scale model to determine disease attributes (such as crack direction, width trend, and leakage status) and generate textual descriptions of the diseases. The geometric information, basic categories, and confidence scores output by the lightweight hybrid deep network are fused with the refined attributes and textual descriptions output by the multimodal large-scale model to form a two-dimensional disease recognition result. The disease identification results include disease category, two-dimensional location (bounding box or pixel-level segmentation mask), detection confidence, and necessary text description, and are indexed by timestamp and camera pose, and passed to step S7 for accurate mapping with the three-dimensional point cloud model.

[0079] Step S7: Map the two-dimensional results to three dimensions and output a three-dimensional model with disease semantic labels;

[0080] Using the recovered absolute-scale camera pose and intrinsic parameters from step S4, the 2D disease identification results are back-projected onto a dense 3D model. The 3D region corresponding to the disease is determined by intersecting the ray with the model, and it is assigned a disease label and color code, outputting a 3D model with disease semantic labels. This process of back-projecting the 2D identification results onto the dense 3D model specifically includes:

[0081] Two-dimensional disease identification result extraction: For each preprocessed two-dimensional panoramic image, the disease identification module outputs a pixel-level mask of the disease. ,in The values ​​are pixel coordinates, and a mask value of 1 indicates that the pixel belongs to the diseased area.

[0082] Pixel ray generation and 3D intersection calculation: Using the camera pose, rays are emitted from the camera's optical center, passing through pixels in the 2D disease area. These rays intersect with multiple 3D units constituting the dense 3D model to determine the target 3D unit associated with the disease area. For each effective pixel in the mask... Using the calibrated metric camera intrinsic parameter matrix and extrinsic parameter matrix The pixel is back-projected into three-dimensional space to form a ray:

[0083] ,

[0084] in, The normalized ray direction in the camera coordinate system. The ray direction vector in the world coordinate system. For the first The translation vector corresponding to each image. The ray intersects the 3D Gaussian splatter model, and the depth of the intersection point between the ray and each Gaussian ellipsoid is calculated using a differentiable renderer (such as the 3D Gaussian Splatting rendering pipeline).

[0085] Gaussian ellipsoid association and label propagation: for each Gaussian ellipsoid traversed by the ray Its central position is The covariance matrix is Opacity is By calculating the distance between the nearest intersection point of the ray and the ellipsoid and the cumulative opacity, the most likely ellipsoid index corresponding to the pixel is determined:

[0086]

[0087] in For rays and ellipsoids The coordinates of the nearest intersection point. The ellipsoid that was hit. The model will be assigned corresponding disease labels (such as cracks, leaks, etc.), the corresponding three-dimensional ellipsoid set will be determined, and the disease labels and color codes will be assigned to them during rendering (such as red for cracks and blue for leaks), thereby generating a three-dimensional visualization model with disease semantic information.

[0088] 3D Disease Model Generation: After traversing all disease pixels in all images and completing label transfer, a 3D Gaussian splash model with disease semantic information is generated. This model not only retains the original geometric and appearance information, but also embeds attributes such as disease category, confidence level, and timestamp into each ellipsoid, supporting subsequent 3D queries, filtering, and statistical analysis.

[0089] Projection consistency optimization: To improve mapping accuracy, a multi-view projection consistency constraint is introduced, using cross-validation of the disease from multiple perspectives. For the same disease area, its mask in multiple images should be projected onto the same set of ellipsoids in the 3D model. If inconsistencies occur, they are corrected by optimizing the ellipsoid labels or adjusting the camera pose.

[0090] Step S8: 3D disease measurement and report output, outputting a 3D model with disease semantic tags and a structured measurement report;

[0091] Geometric measurements of the disease are performed on a metric 3D model labeled with the disease identifiable. Measurement parameters include the length and width of cracks, and the area of ​​planar disease.

[0092] For crack length measurement, the length is obtained by calculating the geodesic path along the crack direction on the 3D model. The main centerline of the 3D point set labeled "crack" is extracted, and the precise length of the crack on the 3D surface is calculated using a geodesic distance algorithm based on the 3D point cloud path.

[0093] For crack width measurement, sampling is performed along the crack skeleton line, and the measurement is obtained by fusing 2D image information and 3D geometric information. Width measurement sampling points are generated at fixed intervals along the crack skeleton line. At each sampling point, the width measurement direction is determined based on the normal vector of the local 3D point cloud. The 3D Euclidean distance between the two boundary points is calculated. At the same time, the 3D width value at this point is cross-validated and calibrated with the corresponding pixel-level crack segmentation mask width in the 2D image using the camera pose.

[0094] For the measurement of the area of ​​planar diseases, the area is obtained by projection and surface correction of the diseased area. A set of Gaussian ellipsoids labeled with the same type of planar disease is obtained, the center point of the ellipsoid is projected onto the local tangent plane, the area of ​​the polygonal region at the projection point boundary is calculated, and then geometric correction is performed based on the average tilt angle of the local surface to convert it into the actual surface area of ​​the 3D model.

[0095] Output a 3D model with semantic tags for defects and a structured measurement report. This step automatically measures key geometric parameters of defects such as cracks, spalling, and leakage on a 3D model with absolute scale, and outputs structured data that can be directly integrated into the operation and maintenance management system.

[0096] Furthermore, this invention also provides a device for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera, used to implement the above method, specifically including:

[0097] Mobile carrier platforms, which can be tracked robots, wheeled robots, or handheld supports, are used to move within caves and carry other equipment.

[0098] At least one panoramic camera unit, which is a 360° panoramic camera (including a fisheye lens and an image sensor), is fixed on a mobile carrier platform and is used to acquire panoramic image sequences of the underground space.

[0099] At least one calibration plate with known physical dimensions is placed on a preset path. The calibration plate has a checkerboard pattern or coded markings (such as AprilTag or ArUco code) printed on its surface and is placed on the preset path. Its key geometric dimensions are the pre-calibrated and known true values.

[0100] The data processing unit receives image data acquired by the panoramic camera unit, executes the above-mentioned method steps, and outputs a three-dimensional semantic model with disease labels and disease quantification parameters.

[0101] At least one auxiliary device: a supplementary lighting assembly for providing active illumination in low-light conditions, and an inertial measurement unit for assisting in pose estimation. This device, equipped with a panoramic camera and calibration board mounted on a mobile platform, combines the above-described method steps with a data processing unit to form a complete visual inspection solution.

[0102] In summary, this invention achieves absolute scale recovery by introducing a scale calibration plate, accurately mapping the two-dimensional disease identification results to a three-dimensional model, realizing the quantitative measurement and analysis of diseases, and solving the problems of the lack of real physical scale in the three-dimensional model and the inability to quantify and measure diseases in the prior art, which has significant progress.

[0103] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera, characterized in that, Includes the following steps: Step S1: Path planning and calibration board setup, outputting the shooting path and scale calibration board setup scheme, the setup scheme includes the spatial position and orientation of at least one scale calibration board with known physical dimensions; Step S2: Panoramic image acquisition and preprocessing. Acquire a sequence of panoramic images of the underground space along the shooting path, preprocess the acquired panoramic image sequence, and output the preprocessed panoramic image sequence. Step S3: Sparse reconstruction and camera parameter estimation. Feature extraction and matching are performed on the preprocessed panoramic image sequence. The sparse 3D point cloud and camera pose of the scene are restored using the motion recovery structure framework. The camera pose and sparse point cloud at a relative scale are obtained and output. Step S4: Scale restoration. Using the known physical size of the scale calibration board, the feature points of the scale calibration board in the preprocessed panoramic image sequence are detected, the global scale factor is calculated, and the global scale factor is used as a constraint to jointly optimize the camera pose and sparse point cloud with relative scale, and output the camera pose and sparse point cloud with absolute physical scale. Step S5: Gaussian splash dense reconstruction. Based on the sparse point cloud with absolute physical scale, a dense three-dimensional model of the underground space is generated using three-dimensional Gaussian splash technology. Step S6: Two-dimensional disease identification. Disease detection and segmentation are performed on the preprocessed panoramic image sequence, and the two-dimensional disease identification results are output. The two-dimensional disease identification results include disease category, bounding box or pixel-level mask. Step S7: Map the two-dimensional results to three dimensions. Using the camera pose with absolute physical scale, back-project the two-dimensional disease identification results onto the dense three-dimensional model. Determine the three-dimensional interval corresponding to the disease by intersecting the ray with the model and assign disease labels. Output a three-dimensional model with disease semantic labels. Step S8: 3D disease measurement and report output. On the 3D model with disease semantic tags, perform geometric measurements on the disease and output the 3D model with disease semantic tags and a structured measurement report.

2. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, The scale calibration board layout scheme includes: generating a sequence of shooting points and a travel path based on the prior physical dimensions of the underground space to be inspected, and outputting the scale calibration board layout scheme, which includes the number, spatial location and orientation of the scale calibration boards.

3. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, Panoramic image acquisition and preprocessing includes: acquiring a sequence of panoramic images of the underground space along a preset shooting path, and flattening, cropping, and correcting distortion of the acquired panoramic images; and generating a noise mask by using a multimodal information fusion mechanism that combines rule thresholding and semantic segmentation models to automatically block out interfering targets in the panoramic images.

4. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, Joint optimization of camera pose and sparse point cloud at relative scale is performed, specifically by incorporating the global scale factor as a constraint into the bundle adjustment optimization process, constructing and solving the joint optimization function, and simultaneously optimizing the camera pose, 3D point coordinates and scale factor to transform the camera pose and sparse point cloud at relative scale into camera pose and sparse point cloud at absolute physical scale.

5. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, The dense 3D model uses the camera pose and sparse point cloud with absolute physical scale as the geometric reference, so that the dense 3D model maintains the metric scale.

6. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, Two-dimensional disease identification includes: using a lightweight hybrid deep network to detect and segment diseases in panoramic images, outputting disease categories, bounding boxes, or pixel-level masks; inputting the image regions defined by the bounding boxes or masks into a multimodal large model for fine-grained analysis to generate textual descriptions of the diseases; and fusing the disease categories, bounding boxes, or pixel-level masks output by the lightweight hybrid deep network with the textual descriptions output by the multimodal large model to form two-dimensional disease identification results.

7. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, The mapping of two-dimensional results to three-dimensional results includes: using the camera pose with absolute physical scale, back-projecting the two-dimensional disease identification results to a dense three-dimensional model, determining the three-dimensional region corresponding to the disease by finding the intersection of rays and Gaussian ellipsoids, introducing multi-view projection consistency constraints, using observations of the disease from multiple perspectives for cross-validation, and outputting a three-dimensional model with disease semantic labels.

8. The method for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera according to claim 1, characterized in that, Geometric measurements include: crack length calculation based on geodesic algorithms, crack width measurement integrating two-dimensional and three-dimensional information, and area calculation of planar defects based on surface correction.

9. A device for three-dimensional reconstruction and disease measurement of underground space based on a panoramic camera, used to implement the method described in any one of claims 1 to 8, characterized in that, include: Mobile carrier platform; At least one panoramic camera unit is fixed on a mobile support platform for acquiring panoramic image sequences of the underground space. At least one scale calibration plate with known physical dimensions is set up on a preset shooting path; a data processing unit receives image data collected by the panoramic camera unit and outputs a three-dimensional semantic model with disease labels and disease quantification parameters; and at least one auxiliary device: a supplementary lighting assembly for supplementary lighting and an inertial measurement unit for auxiliary pose estimation.

Images