A method and system for three-dimensional infrastructure image projection and disease spatial localization

Abstract

A method and system for three-dimensional infrastructure image projection and spatial localization of defects includes: acquiring multi-view real-scene photos, performing aerial triangulation to reconstruct GLB and PLY three-dimensional models, simultaneously calculating camera parameter files to obtain defect data on the photos; parsing camera parameters, converting the aerial triangulation coordinate system to a Three.js coordinate system, creating a view cone, and constructing a virtual camera; traversing all vertices of the GLB three-dimensional model, calculating and generating effective projected UV coordinates and effective marker positions, and rendering the real-scene photos and defect annotation textures onto the surface of the three-dimensional model; calculating the coverage area of ​​the PLY three-dimensional model under the virtual camera view, outputting the UV before and after occlusion, establishing a bidirectional coordinate mapping relationship between the PLY raster and the DZI unfolded map, completing defect matching and accurate texture rendering, and projecting the photo textures and defect annotations onto the DZI unfolded map; displaying the same defect annotation on both the three-dimensional model and the DZI unfolded map view, and finally exporting the defect data-related files.

Description

Technical Field

[0001] This solution relates to the field of 3D infrastructure inspection image processing technology, specifically to a method and system for 3D infrastructure image projection and spatial location of defects. Background Technology

[0002] Large infrastructure structures such as bridges, tunnels, and slopes are prone to structural defects such as cracks, spalling, and water seepage due to long-term exposure to loads and environmental erosion. To ensure structural safety, regular on-site visual inspections are necessary. The current mainstream inspection process involves inspectors taking photos of the structural surface, then using photogrammetry to reconstruct a three-dimensional geometric model of the structure from the images. Finally, defects are manually marked on the two-dimensional photos, and an inspection report is compiled. This approach has significant drawbacks: two-dimensional photos lack the true spatial scale of the structure, and the location of defects cannot be intuitively linked to the physical structure; while the three-dimensional model possesses precise geometric spatial information, it is difficult to accurately attach the defect markings from the photos to the model surface, and it is also impossible to quickly generate standardized two-dimensional defect unfolded diagrams for statistical archiving.

[0003] To address the need for fusion display of 2D images and 3D models, existing technologies, while conducting research on 2D image projection onto 3D model surfaces and texture mapping, still suffer from the following shortcomings: 1. Lack of infrastructure scenario adaptation: Existing photogrammetry projection technologies are mostly geared towards medical applications (CN108694743A), vehicle vision (CN116757958A), or virtual scene generation (CN122415835A). They lack a precise photo-to-3D model projection link driven by intrinsic parameters (focal length, optical center, distortion coefficient) and extrinsic parameters (rotation, translation) obtained from photogrammetric analysis. They also lack refined photogrammetry projection and defect detection solutions for large infrastructure structures such as bridges, tunnels, and slopes. 2. Insufficient photogrammetry projection accuracy: Panoramic sphere projection (CN118781312A) has limited accuracy in high-precision texture mapping of non-panoramic photos and does not support distortion correction and pixel-level projection of single photos. 3. Dual-view linkage without disease annotation: The existing solution cannot achieve automatic alignment and linkage display of the same disease annotation between the 3D model view and the 2D unfolded view. Engineering inspection personnel still need to manually compare between the two views. 4. Lack of lightweight drawing export mechanism: The existing solution does not support batch projection of a large number of disease annotation photos onto the unfolded view and exporting them as interactive SVG and structured CSV, which is not convenient for generating and statistically analyzing inspection reports.

[0004] Therefore, for the appearance inspection of infrastructure such as bridges, tunnels, and slopes, there is an urgent need for a high-precision 3D projection method based on real-scene photos using photogrammetric camera parameters, and to achieve dual-view linkage between 3D models and 2D unfolded diagrams with defect annotations, as well as batch defect production. Figure 1 Integrated output processing scheme. Summary of the Invention

[0005] This solution aims to overcome at least one deficiency in existing technologies and provides a method and system for three-dimensional infrastructure image projection and spatial location of defects, used to achieve linkage between two views of three-dimensional models and two-dimensional unfolded maps with defect annotations, and batch production of defects. Figure 1 Integrated output and other effects.

[0006] To solve the above-mentioned technical problems, the following technical solution is adopted: On the one hand, this solution provides a method for three-dimensional infrastructure image projection and spatial localization of defects, the method comprising the following steps: S1: Data acquisition and preprocessing are performed. Multi-view real-scene photos of the infrastructure structure surface are collected, and aerial triangulation is performed on the real-scene photos to reconstruct GLB and PLY 3D models. At the same time, the camera parameter files corresponding to each real-scene photo are calculated, and the defect data marked on the real-scene photos are obtained. S2: Perform coordinate system adaptation and virtual camera construction, parse the camera parameters, convert the aerial coordinate system to the Three.js standard coordinate system, and create a view cone based on the parsed camera intrinsic and extrinsic parameters to construct the virtual camera; S3: Perform 3D model projection, traverse all vertices of the GLB 3D model according to the constructed virtual camera, calculate and generate the effective projection UV coordinates and effective marker positions of each vertex, and render and fit the real scene photo texture and disease annotation texture onto the surface of the 3D model. S4: Project the DZI unfolded map, calculate the coverage area of ​​the PLY 3D model in the virtual camera view, output the UV data before and after occlusion, then establish a two-way coordinate mapping relationship between the PLY grid and the DZI unfolded map, and complete the disease matching and accurate texture drawing based on the UV data before and after occlusion, projecting the photo texture and disease annotations onto the DZI unfolded map; S5: Perform dual-view disease linkage visualization and export. The same disease annotation is displayed on the 3D model view through ShaderMaterial overlay and on the DZI unfolded view through SVG layer overlay. Finally, the processed disease data is exported in multiple standardized formats, and the exported file data is interactive SVG, structured CSV and 3D disease marker JSON file.

[0007] This solution targets the inspection and maintenance of large-scale infrastructure such as bridges, tunnels, and slopes. Through a multi-step collaborative 3D projection and defect localization method, it effectively solves the technical defects of traditional infrastructure defect detection, such as texture projection misalignment, defect matching omissions, limitations of single-view display, and uniform result formats. Relying on aerial triangulation reconstruction, it obtains GLB and PLY dual models to meet the dual scene requirements of 3D rendering and 2D unfolding. Through coordinate system transformation and virtual camera construction, it ensures the accuracy of 3D projection calculation. At the same time, it innovatively adopts a dual UV differential calculation mechanism before and after occlusion, which avoids defect mismatch problems caused by model occlusion and ensures texture projection fitting accuracy. In addition, it combines affine transformation and rasterization rendering technology to achieve high-precision projection of 2D unfolded images. Coupled with the dual-view linkage display mode of ShaderMaterial material 3D overlay and SVG layer 2D overlay, it achieves spatial alignment and synchronous visualization of defect annotations. Ultimately, it automatically outputs three types of standardized deliverables: interactive vector graphics, structured data tables, and 3D marked files. This takes into account the needs of visual verification, data statistical archiving, and 3D scene reproduction, significantly improving the spatial positioning accuracy, display effect, and standardization of output of infrastructure defects.

[0008] Preferably, in step S1, the aerial triangulation process is as follows: the multi-view real-scene photos are reconstructed into three dimensions using the motion recovery structure algorithm to obtain a GLB three-dimensional model for three-dimensional display and ray occlusion detection and a PLY three-dimensional model for DZI unfolded coordinate mapping.

[0009] By using a motion recovery structure algorithm to reconstruct 3D images, two distinct types of 3D models are generated. The GLB 3D model is specifically designed for 3D display and ray occlusion detection, ensuring accurate and reliable 3D projection occlusion determination. The PLY 3D model is specifically adapted to DZI unfolded map coordinate mapping scenarios, achieving precise projection matching between 2D unfolded textures and lesions. This dual-model approach effectively avoids the accuracy loss caused by a single model handling multiple scenarios, improving the overall projection and lesion localization accuracy from the source.

[0010] Preferably, in step S2, creating the view cone includes: calculating the field of view (FOV) based on the intrinsic parameters obtained from the analysis, and filtering and screening disabled cameras and uncalibrated cameras; calculating the position and orientation of the camera in the world coordinate system based on the extrinsic parameters, and defining the spatial range of the view cone.

[0011] The above steps combine camera intrinsic and extrinsic parameters to achieve accurate construction of the view frustum and pre-filtering of invalid data. By solving the field of view (FOV) using intrinsic parameters, disabled and uncalibrated cameras are automatically eliminated, thus pre-screening invalid calculation sources and reducing redundant calculations and erroneous projections. Simultaneously, relying on camera extrinsic parameters, the camera's spatial pose is accurately calculated, precisely defining the effective coverage area of ​​the view frustum and standardizing the calculation boundaries for subsequent 3D projection. This effectively avoids projection deviations and calculation errors caused by out-of-field-of-view or invalid cameras, ensuring the accuracy and stability of the virtual camera projection reference. This provides a reliable foundation for subsequent model visible area selection and accurate UV calculation.

[0012] Preferably, step S3 includes: sequentially performing backface culling, perspective projection, distortion correction, UV normalization, UV boundary verification, and ray occlusion detection on the vertices of the GLB 3D model, filtering out valid projected UV coordinates and valid marker positions, and performing texture and hazard annotation bonding rendering only on valid vertices.

[0013] This step performs multi-stage preprocessing and verification on the vertices of the GLB 3D model, sequentially completing backface culling, distortion correction, and ray occlusion detection. It then filters out valid projected UV coordinates and valid markers, rendering only on the valid vertices. This eliminates projection errors caused by backfaces, occluded areas, and distortion, reduces rendering overhead for invalid vertices, improves the accuracy of texture and hazard annotation alignment, and simultaneously reduces rendering computational costs.

[0014] Preferably, the distortion correction is a full-parameter forward coupling correction method that integrates radial distortion, tangential distortion, and affine distortion. It performs normalization deviation compensation and denormalization restoration on the projected pixel coordinates, eliminates the projection offset caused by lens distortion, and obtains high-precision distortion-free projection coordinates.

[0015] By integrating radial, tangential, and affine multi-angle distortions to achieve full-parameter coupling correction, and performing normalized deviation compensation and denormalized restoration processing of projected pixel coordinates, the projection offset caused by various lens distortions is fully offset, high-precision projection coordinates are obtained, the pixel alignment accuracy of 2D photos projected onto 3D models is improved, the misalignment of texture and defect annotations is reduced, and accurate basic data is provided for subsequent coordinate mapping and dual-view linkage.

[0016] Preferably, in step S4, establishing the bidirectional coordinate mapping relationship between the PLY grid and the DZI unfolded image includes: based on the row and column index of the PLY grid cell, the overall grid size, and the resolution parameters of the DZI unfolded image, converting the three-dimensional PLY grid coordinates to the two-dimensional DZI unfolded image pixel coordinates through a linear mapping method, and constructing a coordinate correspondence relationship that can be queried bidirectionally.

[0017] Based on the PLY raster row and column index, size, and DZI unfolded image resolution, linear mapping is used to realize the conversion between 3D raster coordinates and 2D pixel coordinates, and a bidirectional queryable coordinate association system is constructed. This can unify the coordinate reference of the 3D model surface and the 2D unfolded image, ensuring that the disease annotation and texture data can accurately correspond in the two views. The bidirectional mapping supports fast conversion between positive and negative coordinates, which facilitates the synchronization of 3D modifications to the 2D unfolded image. It can also realize the reverse positioning of 2D annotations to the 3D model, providing stable and reliable coordinate support for the linkage and alignment of diseases in the two views.

[0018] Preferably, step S4 includes: projecting the photo texture onto the DZI unfolded image using affine transformation based on the UV after occlusion; performing AABB overlap detection on the UV range of the disease and the PLY grid unit based on the UV before occlusion; establishing the correspondence between the disease and the unfolded image unit; and projecting the disease label onto the DZI unfolded image.

[0019] By distinguishing between UV after occlusion and UV before occlusion and processing texture projection and disease matching separately, affine transformation is used to complete texture projection, and AABB overlap detection is used to associate diseases with mesh cells. The above separation of texture rendering and disease matching calculation logic takes into account both texture display accuracy and disease identification completeness, avoids disease omission caused by occlusion, establishes the binding relationship between diseases and unfolded map cells, ensures accurate positioning of disease annotations in 2D unfolded map, supports disease linkage alignment between 3D model and DZI unfolded map, and improves the reliability of infrastructure disease annotation.

[0020] Preferably, the step of projecting the photo texture onto the DZI unfolded image by affine transformation based on the occluded UV includes: calculating a three-point affine transformation matrix for the four effective UVs at the four corners of each PLY grid cell, and then performing Canvas / WebGL rasterization on the PLY raster texture area after affine transformation correction, thereby projecting the local texture of the photo onto the DZI unfolded image.

[0021] This method utilizes the effective UVs at the four corners of the PLY mesh to solve the three-point affine transformation matrix. After correcting the texture deformation, it uses Canvas / WebGL to complete the GPU parallel rasterization rendering, projecting the photo texture onto the DZI unfolded image. This can eliminate the problem of texture stretching and distortion on curved surfaces, improve texture fitting accuracy, improve rendering efficiency by relying on GPU parallel computing, and ensure the real-time rendering of texture unfolded images for large-scale infrastructure models.

[0022] Preferably, step S5 includes: automatically deduplicating, performing connectivity analysis and OBB bounding box calculation on all disease annotations on the DZI unfolded diagram, and exporting the optimized disease data as interactive SVG, structured CSV and three-dimensional disease marker JSON files.

[0023] This step performs automatic deduplication, connectivity analysis, and OBB bounding box calculation on the disease annotations, achieving standardization and optimization of disease data. It simultaneously exports interactive SVG, structured CSV, and 3D disease-marked JSON files in multiple formats, catering to the needs of visual browsing, data statistical archiving, and 3D scene reuse, reducing manual processing workload, and adapting to infrastructure inspection report preparation and long-term operation and maintenance archiving.

[0024] On the other hand, this solution also provides a system for three-dimensional infrastructure image projection and spatial localization of defects, the system comprising: The data acquisition and preprocessing module, configured in the data acquisition and preprocessing layer, is used to acquire real-scene photos, reconstruct GLB and PLY 3D models through aerial triangulation, solve camera parameter files, and obtain JSON format disease annotation data. The projection calculation and mapping module, configured in the projection calculation and mapping layer, is used to complete camera parameter parsing, conversion between the aerial coordinate system and the Three.js standard coordinate system, creation of camera view cone and construction of virtual camera, and to perform projection calculation, distortion correction and occlusion detection on the vertices of the GLB model, generating the effective projected UV coordinates and effective flags of each vertex. The coordinate mapping and unfolding module, configured in the coordinate mapping and unfolding layer, is used to construct the bidirectional coordinate mapping relationship between the PLY grid and the DZI unfolded image, solve the three-point affine transformation matrix of the PLY grid cell, complete the AABB matching and binding of the disease and the grid cell, and establish the correspondence between the disease and the unfolded image cell. The visualization and export module, configured in the visualization and export layer, includes a dual-view visualization unit and a multi-format export unit. It is used to realize the linkage display of disease in dual views of 3D view and DZI unfolded diagram, optimize the disease data, and finally export the results as interactive SVG, structured CSV, and 3D disease marker JSON standardized output files.

[0025] This system adopts a layered modular architecture, targeting infrastructure defect detection scenarios such as bridges, tunnels, and slopes. It overcomes the shortcomings of existing technologies, such as lack of engineering adaptability and fragmented functions. Each module has a clear responsibility and works collaboratively to fully realize the entire process of 3D infrastructure image projection and defect spatial localization. The data acquisition and preprocessing module completes image reconstruction, model building, and defect data collection; the projection calculation and mapping module unifies the coordinate reference, combines distortion correction and occlusion detection to output accurate and effective UV data, ensuring pixel-level projection accuracy; the coordinate mapping and unfolding module establishes a two-way coordinate association between the 3D raster and the DZI unfolded image, and achieves accurate texture and defect mapping through affine transformation and AABB matching; the visualization and export module achieves dual-view defect linkage and alignment, automatically optimizes defect data, and outputs multiple standardized files. The entire system solves the pain points of traditional solutions, such as the inability to link dual views and the inconvenience of exporting results, reducing the workload of manual comparison and processing, and facilitating defect statistics, result archiving, and test report preparation.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. High-precision projection Based on the complete camera parameters (including radial / tangential / affine distortion coefficients) obtained from photogrammetry analysis, pixel-level accurate projection from 2D photographs to 3D models is achieved, with the geometric error of the projected texture being less than 1 pixel.

[0027] 2. Automatic linkage between dual views The same lesion annotation is automatically aligned on the 3D model surface and the DZI unfolded image, eliminating the need for manual comparison. Modifications to lesions on the 3D model can be synchronized to the unfolded image via UV mapping, and vice versa.

[0028] 3. Multiple photo overlay and deduplication It supports overlaying projections of multiple photos at the same 3D model location; it automatically removes duplicates through a strategy that minimizes UV correction, preserving the best projection quality.

[0029] 4. Lightweight interactive export Export SVG as an interactive web page (supporting zoom, pan, and thumbnail navigation), and CSV as a structured OBB statistical table, adapting to the needs of test report generation and archiving.

[0030] 5. Performance optimization for million-level triangular networks By employing mechanisms such as AABB localized filtering, BVH occlusion detection localization, coverage cache reuse, and UV spatial index sparsification, smooth real-time projection interaction is achieved on a million-triangle mesh model.

[0031] 6. Multi-projector group support The projectionGroups mechanism supports multiple independent models (such as left and right holes, up and down holes) under the same case, with each group having independent photos, XML, GLB / PLY, and DZI configurations. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0033] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0034] Figure 3 This is a schematic diagram of the system flow of the present invention.

[0035] Figure 4 This is a schematic diagram of the derived results of the present invention. Detailed Implementation

[0036] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate the following embodiments, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0037] Example 1 like Figure 1 As shown in the figure, this embodiment provides a method for three-dimensional infrastructure image projection and spatial localization of defects, the method including the following steps: Step S1: Data acquisition and preprocessing. Collect multi-view real-scene photos of the infrastructure structure surface, perform aerial triangulation on the real-scene photos to reconstruct GLB and PLY 3D models, and simultaneously calculate the camera parameter file corresponding to each real-scene photo. Also, obtain the defect data marked on the real-scene photos. Preferably, in step S1, the aerial triangulation process involves: reconstructing the multi-view real-scene photographs into three dimensions using a motion recovery structure algorithm to obtain a GLB 3D model for 3D display and ray occlusion detection, and a PLY 3D model for DZI unfolded coordinate mapping. The GLB 3D model is a GLB format triangular mesh 3D model, and the PLY 3D model is a PLY format regular rasterized model.

[0038] Furthermore, the camera parameter file in this embodiment includes: intrinsic parameter data such as focal length f, principal point offset cx, cy, radial / tangential / affine distortion coefficients, etc., and extrinsic parameter data such as rotation matrix R and translation vector t in the world coordinate system. For defect data, in this embodiment, information such as the coordinates, category, and color of each defect is collected by storing defect polygons (cracks, leaks, peeling, etc.) marked on two-dimensional photographs by inspectors.

[0039] This approach utilizes a motion recovery structure algorithm to reconstruct 3D images from real-world scenes, generating two distinct types of 3D models with clearly defined functions. The GLB model is specifically designed for 3D display and ray occlusion detection, ensuring accurate and reliable 3D projection occlusion assessment. The PLY model is specifically adapted to DZI unfolded map coordinate mapping scenarios, achieving precise projection alignment between 2D unfolded textures and lesions. This dual-model approach effectively avoids the accuracy loss caused by a single model handling multiple scenarios, improving overall projection and lesion localization accuracy from the source.

[0040] Step S2: Perform coordinate system adaptation and virtual camera construction. Parse the camera parameters, convert the aerial coordinate system to the Three.js standard coordinate system, and create a view cone based on the parsed camera intrinsic and extrinsic parameters to construct the virtual camera. Preferably, in step S2, creating the view cone includes: calculating the field of view (FOV) based on the intrinsic parameters obtained from the analysis, and filtering and screening disabled cameras and uncalibrated cameras; calculating the position and orientation of the camera in the world coordinate system based on the extrinsic parameters, and defining the spatial range of the view cone.

[0041] Specifically, the parsing of camera parameters includes parsing camera pose XML (document format) and photo document XML (frame format), and processing the ID offset accumulation of multiple file pairs, calibration parameter integrity check and aerial coordinate system → Three.js transformation, i.e.: p_three.js = (x_ms, z_ms, -y_ms).

[0042] Next, the field of view (FOV) is calculated based on the intrinsic parameters (focal length f, optical center cx, cy) obtained from the analysis, filtering out disabled cameras and uncalibrated cameras; the camera's position and orientation in the world coordinate system are calculated based on the extrinsic parameters (rotation matrix in the world coordinate system). The formula for calculating the FOV is: FOV = 2×arctan(H / (2f)) × 180 / π, where H is the image height and f is the focal length. The formula for calculating the camera orientation is: d_look = (r 02 , r 22 , -r 12) ^T / ||·||,d_up = (-r 01, -r 21 , r 11 )^T / ||·||, where d_look is the unit vector representing the camera's viewing direction in the world coordinate system, used to determine the view frustum direction; d_up is the vertical upward unit vector representing the camera's position in the world coordinate system, used to correct camera pitch and tilt to avoid image flip distortion, r ij represents the matrix element of the rotation matrix R in the world coordinate system.

[0043] Thus, the above steps combine camera intrinsic and extrinsic parameters to achieve accurate construction of the view frustum and pre-filtering of invalid data. By solving the field of view (FOV) using intrinsic parameters, disabled and uncalibrated cameras are automatically eliminated, and invalid calculation sources are preemptively screened, reducing redundant calculations and erroneous projections. Simultaneously, relying on camera extrinsic parameters, the camera's spatial pose is accurately calculated, precisely defining the effective coverage area of ​​the view frustum and standardizing the calculation boundaries for subsequent 3D projection. This effectively avoids projection deviations and calculation errors caused by out-of-field-of-view or invalid cameras, ensuring the accuracy and stability of the virtual camera projection reference, and providing a reliable foundation for subsequent model visible area selection and accurate UV calculation.

[0044] Step S3: Perform 3D model projection. Based on the constructed virtual camera, traverse all vertices of the GLB 3D model, calculate and generate the effective projection UV coordinates and effective marker positions of each vertex, and render and fit the real scene photo texture and disease annotation texture onto the surface of the 3D model. Preferably, step S3 includes: sequentially performing backface culling, perspective projection, distortion correction, UV normalization, UV boundary verification, and ray occlusion detection on the vertices of the GLB 3D model, filtering out valid projected UV coordinates and valid marker positions, and performing texture and hazard annotation bonding rendering only on valid vertices.

[0045] The perspective projection formula is: px = -f·xcam / zcam + cx, py = -f·ycam / zcam + cy, where (xcam,ycam,zcam) are the local camera coordinates of the vertex in the camera coordinate system, cx and cy are the principal point offsets, and (px,py) are the ideal (distortion-free) projected pixel coordinates. Next, distortion correction is performed. In this embodiment, the distortion correction is a fully parametric forward coupling correction method that integrates radial distortion, tangential distortion, and affine distortion. Normalization deviation compensation and denormalization restoration are performed on the projected pixel coordinates to eliminate projection offset caused by lens distortion, resulting in high-precision distortion-free projection coordinates. The forward distortion model formula is as follows: Normalized coordinates: x = (px - cx) / f, y = (py - cy) / f, r² = x² + y². = x·(1+k1r²+k2r 4 +k3r 6 ) + 2p1xy + p2(r²+2x²) + b1r², = y·(1+k1r²+k2r 4 +k3r 6 ) + p1(r²+2y²) + 2p2xy + b2r², Anti-normalization: x = ·f + cx, y = ·f + cy, in,( x, y) represents the pixel coordinates after distortion correction, k1, k2, k3 are the radial distortion coefficients, p1, p2 are the tangential distortion coefficients, b1, b2 are the affine distortion coefficients, and r is the radial distance from the pixel to the origin on the normalized plane. , () represents the normalized image plane coordinates after distortion correction.

[0046] The UV normalization formula is: u = x / W, v = 1 - y / H, where W is the width of the photo and H is the height of the photo.

[0047] Preferably, in this embodiment, the ray obstruction detection uses Raycaster ray detection to perform obstruction detection.

[0048] Therefore, the above steps perform multi-stage preprocessing and verification on the vertices of the GLB 3D model, sequentially completing backface culling, distortion correction, and ray occlusion detection, filtering out valid projected UV coordinates and valid marker positions, and rendering only on valid vertices. This eliminates projection errors caused by backfaces, occluded areas, and distortion, reduces rendering overhead for invalid vertices, improves the accuracy of texture and defect annotation, and reduces rendering computational power consumption.

[0049] Step S4: Project the DZI unfolded image, calculate the coverage area of ​​the PLY 3D model in the virtual camera view, output the UV data before and after occlusion, then establish a two-way coordinate mapping relationship between the PLY grid and the DZI unfolded image, and based on the UV data before and after occlusion, complete the disease matching and accurate texture drawing respectively, and project the photo texture and disease annotations onto the DZI unfolded image; Preferably, in step S4, establishing the bidirectional coordinate mapping relationship between the PLY grid and the DZI unfolded image includes: based on the row and column index of the PLY grid cell, the overall grid size, and the resolution parameters of the DZI unfolded image, converting the three-dimensional PLY grid coordinates to the two-dimensional DZI unfolded image pixel coordinates through a linear mapping method, and constructing a coordinate correspondence relationship that can be queried bidirectionally.

[0050] Specifically, the mapping relationship calculation formula is: xdzi = xply + (row+0.5) / Nrows × Wdzi, ydzi = yply + (col+0.5) / Ncols × Hdzi, where (xdzi,ydzi) are the coordinates of the DZI unfolded image, (row,col) are the PLY raster cell indices, Wdzi is the width of the DZI unfolded image, Hdzi is the height of the DZI unfolded image, Nrows is the number of PLY raster rows (streamlines), and Ncols is the number of PLY raster columns (traces).

[0051] By leveraging the PLY raster row and column index, size, and DZI unfolded image resolution, linear mapping is used to convert 3D raster coordinates to 2D pixel coordinates, constructing a bidirectional queryable coordinate association system. This unifies the coordinate reference between the 3D model surface and the 2D unfolded image, ensuring that the disease annotations and texture data can accurately correspond in both views. The bidirectional mapping supports rapid conversion between positive and negative coordinates, facilitating the synchronization of 3D modifications to the 2D unfolded image. It can also enable the reverse positioning of 2D annotations to the 3D model, providing stable and reliable coordinate support for the linkage and alignment of diseases in both views.

[0052] Further preferably, step S4 includes: projecting the photo texture onto the DZI unfolded image using affine transformation based on the UV after occlusion; performing AABB overlap detection on the UV range of the lesion and the PLY mesh unit based on the UV before occlusion; establishing the correspondence between the lesion and the unfolded image unit; and projecting the lesion annotation onto the DZI unfolded image. Specifically, in this embodiment, AABB overlap detection is performed on the four corner ranges represented by the lesion pixel coordinates and the UV range of the PLY unit to establish the correspondence between the lesion and the unfolded image unit.

[0053] Therefore, in this embodiment, by distinguishing between UV after occlusion and UV before occlusion to process texture projection and disease matching respectively, affine transformation is used to complete texture projection, and AABB overlap detection is used to associate diseases with mesh cells. The above separation of texture rendering and disease matching calculation logic takes into account both texture display accuracy and disease identification completeness, avoids disease omission caused by occlusion, establishes the binding relationship between diseases and unfolded map cells, ensures accurate positioning of disease annotations in the two-dimensional unfolded map, supports disease linkage alignment between the three-dimensional model and the DZI unfolded map, and improves the reliability of infrastructure disease annotation.

[0054] More preferably, the step of projecting the photo texture onto the DZI unfolded image by affine transformation based on the occluded UV includes: calculating a three-point affine transformation matrix for the four effective UVs at the four corners of each PLY grid cell, and then performing Canvas / WebGL rasterization on the PLY raster texture area after affine transformation correction, thereby projecting the local texture of the photo onto the DZI unfolded image.

[0055] This method utilizes the effective UVs at the four corners of the PLY mesh to solve the three-point affine transformation matrix. After correcting the texture deformation, it uses Canvas / WebGL to complete the GPU parallel rasterization rendering, projecting the photo texture onto the DZI unfolded image. This can eliminate the problem of texture stretching and distortion on curved surfaces, improve texture fitting accuracy, improve rendering efficiency by relying on GPU parallel computing, and ensure the real-time rendering of texture unfolded images for large-scale infrastructure models.

[0056] Step S5: Perform dual-view disease linkage visualization and export. Display the same disease annotation on the 3D model view using ShaderMaterial overlay and on the DZI unfolded view using SVG layer overlay. Finally, export the processed disease data as interactive SVG, structured CSV and 3D disease marker JSON files.

[0057] Specifically, in this embodiment, for the 3D model projection view: a photo texture or disease mask canvas is overlaid onto the 3D model surface using ShaderMaterial, supporting transparency adjustment, multiple photo overlay, and interactive switching. For the DZI unfolded image projection view: based on OpenSeadragon's SVG overlay, the projection result is overlaid onto the DZI unfolded image as an SVG image element, supporting independent management and switching of multiple photos and multiple disease layers. This allows the same disease annotation to be projected and displayed simultaneously in both the 3D model surface (ShaderMaterial) and the DZI unfolded image (SVG overlay). Both views use the same set of camera parameters and UV mapping to ensure spatial consistency, and the disease projection uses an independent layer management system (i.e., each photo's photo / disease layer has an independent key), thus enabling multi-photo comparison without mutual deletion.

[0058] Preferably, step S5 includes: automatically deduplicating, performing connectivity analysis and OBB bounding box calculation on all disease annotations on the DZI unfolded diagram, and exporting the optimized disease data as interactive SVG, structured CSV and three-dimensional disease marker JSON files.

[0059] Specifically, in this embodiment, from the user-selected photo set, the projection area of ​​the lesion on the unfolded image is calculated image by image. After deduplication (i.e., only one photo with minimal UV distortion is retained at the same location), connectivity analysis, and OBB bounding box calculation, it is exported as an interactive SVG (containing data such as mesh base map, lesion mask, legend, and OBB outline), a structured CSV (containing data such as OBB four-corner coordinates, area, and source photo), and a 3D lesion JSON (which can be read back to the 3D model display). Thus, it is exported as follows: Figure 4 The disease display effect shown implements a controlled UV compensation mechanism for disease edges in photos, high-precision navigation of SVG thumbnails, and double-click linkage between the CSV floating list and the 3D view. The exported SVG is an interactive web page (supporting zooming, panning, and thumbnail navigation), and the CSV is a structured OBB statistical table, which is adapted to the needs of test report generation and archiving, thus achieving lightweight interactive export.

[0060] The above steps, through automatic deduplication, connectivity analysis, and OBB bounding box calculation of disease annotations, achieve standardized optimization of disease data. Multiple formats can be simultaneously exported: interactive SVG, structured CSV, and 3D disease-marked JSON files. This caters to the needs of visual browsing, data statistical archiving, and 3D scene reuse, reducing manual processing workload and adapting to infrastructure inspection report preparation and long-term maintenance archiving.

[0061] Therefore, the solution described in this embodiment is geared towards the inspection and maintenance scenarios of large-scale infrastructure such as bridges, tunnels, and slopes. Through a multi-step collaborative 3D projection and defect localization method, it effectively solves the technical defects of traditional infrastructure defect detection, such as texture projection misalignment, defect matching omissions, single-view display limitations, and uniform result formats. Relying on aerial triangulation reconstruction, it obtains GLB and PLY dual models to meet the dual scene requirements of 3D rendering and 2D unfolding. Through coordinate system transformation and virtual camera construction, it ensures the accuracy of 3D projection calculation. At the same time, it innovatively adopts a dual UV differential calculation mechanism before and after occlusion, which avoids defect mismatch problems caused by model occlusion and ensures texture projection fitting accuracy. In addition, it combines affine transformation and rasterization rendering technology to achieve high-precision projection of 2D unfolded images. Coupled with the dual-view linkage display mode of ShaderMaterial material 3D overlay and SVG layer 2D overlay, it achieves spatial alignment and synchronous visualization of defect annotations. Ultimately, it automatically outputs three types of standardized deliverables: interactive vector graphics, structured data tables, and 3D marked files. This takes into account the needs of visual verification, data statistical archiving, and 3D scene reproduction, significantly improving the spatial positioning accuracy, display effect, and standardization of output of infrastructure defects.

[0062] Example 2 like Figure 2As shown, based on the method for three-dimensional infrastructure image projection and spatial location of defects described in Embodiment 1, this embodiment provides a system for three-dimensional infrastructure image projection and spatial location of defects, the system comprising: The data acquisition and preprocessing module, configured in the data acquisition and preprocessing layer, is used to acquire real-scene photos, reconstruct GLB and PLY 3D models through aerial triangulation, solve camera parameter files, and obtain JSON format disease annotation data. The projection calculation and mapping module, configured in the projection calculation and mapping layer, includes the camera parameter parser XMLParser, the 3D model photo projector PhotoProjector, and the coverage area and UV calculator CoverageCalculator. It is used to complete camera parameter parsing, conversion between the aerial coordinate system and the Three.js standard coordinate system, creation of camera view cone and construction of virtual camera, and to perform projection calculations, distortion correction, and occlusion detection on the vertices of the GLB model, generating the effective projected UV coordinates and effective flag bits of each vertex. The coordinate mapping and unfolding module, configured in the coordinate mapping and unfolding layer, includes the PLY-DZI coordinate mapper DZIProjector, the affine transformation engine AffineTransformer, and the disease marker-UV matcher DiseaseMapper. It is used to construct the bidirectional coordinate mapping relationship between the PLY grid and the DZI unfolded map, solve the three-point affine transformation matrix of the PLY grid cell, complete the AABB matching and binding of the disease and the grid cell, and establish the correspondence between the disease and the unfolded map cell. The visualization and export module, configured in the visualization and export layer, includes a dual-view visualization unit and a multi-format export unit. It is used to realize the linkage display of disease in dual views of 3D view and DZI unfolded diagram, optimize the disease data, and finally export the results as interactive SVG, structured CSV, and 3D disease marker JSON standardized output files.

[0063] Specifically, in this embodiment, the system is used to perform a tunnel inspection project, and the specific system flow is as follows: Figure 3 As shown, the inspectors took photos of the tunnel lining from multiple angles, obtaining approximately 8,000 images. Aerial triangulation was then performed using photogrammetry software to reconstruct a 3D model (approximately 2 million triangles in a GLB triangulation network and 640×128 PLY grid cells).

[0064] First, the JSON configuration file is loaded, and the PhotoProjection configuration section and projectionGroups array are identified. The correspondence between the photo paths, XML files, GLB / PLY models, and DZI unfolded diagrams of each projection group is determined. In this embodiment, projection groups containing two tunnels, one on the left and one on the right, are configured, each corresponding to an independent XML, GLB, PLY model, and DZI unfolded diagram. Then, it is determined whether PhotoProjection is included. If not, the system process ends due to no photo projection; if so, the process proceeds to the next step, parsing projectionGroups to determine the correspondence between projection groups.

[0065] Next, the camera parameters are parsed. The camera pose XML and photo document XML are parsed using an XMLParser to determine if they are multiple file pairs. If so, ID offset accumulation is performed; otherwise, single-pair parsing is performed, thus constructing a camera / sensor / component index table. This supports the accumulation and merging of multiple file pairs and ID offsetting—multiple XML files may start from the same ID number (0), and the _idOffset mechanism ensures the global uniqueness of sensors, cameras, and components. In this embodiment, when parsing camera parameters, the IDs of the XML files in both tunnels start from 0 and are automatically adjusted to globally unique IDs through the ID offset mechanism.

[0066] Then, a view cone is created. In this embodiment, approximately 8,000 photos pass all filter checks (discarding disabled cameras, uncalibrated sensors, and photos without pose data).

[0067] The user then selects a tunnel crack photo from the filtered images. The system uses the `resolveProjectionGroup` function in the 3D model photo projector to determine the projection target group (a single group is returned directly; multiple groups are determined through AABB ray detection or XML path matching), and then performs the corresponding type of projection. In this embodiment, AABB ray detection automatically matches the appropriate projection group. Thus, through the `projectionGroups` configuration and the `resolveProjectionGroup` function, in cases containing multiple independent 3D models (such as left and right tunnels, up and down tunnels), the system automatically determines the projection group to which each photo belongs. Furthermore, combining AABB ray detection and XML path matching to determine the projection target avoids projecting onto the wrong model in multi-model scenarios.

[0068] Next, the projection calculation and mapping module, coordinate mapping and unfolding module, visualization and export module are used to perform 3D model photo projection, DZI unfolded image photo projection, and disease export.

[0069] Specifically, in the 3D model projection: traversing the vertices of approximately 2 million triangles, after AABB localization filtering (range 10m), approximately 300,000 vertices are actually calculated, distortion correction and UV projection are completed, and a ShaderMaterial overlay layer is generated, which takes approximately 200ms.

[0070] In the DZI unfolded image projection: the coverage area is calculated by computeCoverageWithUV, affine transformation is performed unit by unit (PLY), then Canvas / WebGL rasterization is performed, and finally the Canvas is rendered and overlaid on the SVG overlay.

[0071] Finally, export as follows: Figure 4 The diagram shows the projection results of the defects: Defect annotation data (including 1 crack and 1 water seepage) from the same photograph is loaded, generating a defect mask Canvas, which is then displayed on the 3D model and unfolded diagram via a projection link. This achieves a pixel deviation of less than 1 pixel between the 3D model's projected texture and the original photograph at key feature points; the same crack annotation is perfectly aligned on both the 3D model and unfolded diagram without manual adjustment; Defect export from unfolded diagram: SVG (including mesh base map, defect mask, legend, and OBB) is exported from the unfolded diagram of the left tunnel, with a file size of approximately 3MB, supporting browser zoom navigation; CSV (containing defect number, OBB four-corner coordinates, area, and source photograph) is also exported; 3D defect JSON export: The deduplicated defect markers are expanded by seed → connected component recovery → 3D ray hit generation, and displayed as wireframes on the 3D model, with depthTest:true to avoid clipping.

[0072] Therefore, this system adopts a layered modular architecture, targeting infrastructure defect detection scenarios such as bridges, tunnels, and slopes. It overcomes the shortcomings of existing technologies, such as lack of engineering adaptability and fragmented functions. Each module has a clear responsibility and works collaboratively to fully realize the entire process of 3D infrastructure image projection and defect spatial localization. The data acquisition and preprocessing module completes image reconstruction, model building, and defect data collection; the projection calculation and mapping module unifies the coordinate reference, combines distortion correction and occlusion detection to output accurate and effective UV data, ensuring pixel-level projection accuracy; the coordinate mapping and unfolding module establishes a two-way coordinate association between the 3D raster and the DZI unfolded image, and achieves accurate texture and defect mapping through affine transformation and AABB matching; the visualization and export module realizes dual-view defect linkage alignment, automatically optimizes defect data, and outputs multiple standardized files. The entire system solves the pain points of traditional solutions, such as the inability to link dual views and the inconvenience of exporting results, reducing the workload of manual comparison and processing, and facilitating defect statistics, result archiving, and test report preparation.

[0073] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for three-dimensional infrastructure image projection and spatial localization of defects, characterized in that, The method includes the following steps: S1: Data acquisition and preprocessing are performed. Multi-view real-scene photos of the infrastructure structure surface are collected, and aerial triangulation is performed on the real-scene photos to reconstruct the GLB 3D model and PLY 3D model. At the same time, the camera parameter file corresponding to each real-scene photo is calculated, and the disease data marked on the real-scene photos is obtained. S2: Perform coordinate system adaptation and virtual camera construction, parse the camera parameters, convert the aerial coordinate system to the Three.js standard coordinate system, and create a view cone based on the parsed camera intrinsic and extrinsic parameters to construct the virtual camera; S3: Perform 3D model projection, traverse all vertices of the GLB 3D model according to the constructed virtual camera, calculate and generate the effective projection UV coordinates and effective marker positions of each vertex, and render and fit the real scene photo texture and disease annotation texture onto the surface of the 3D model. S4: Project the DZI unfolded map, calculate the coverage area of ​​the PLY 3D model in the virtual camera view, output the UV data before and after occlusion, then establish a two-way coordinate mapping relationship between the PLY grid and the DZI unfolded map, and complete the disease matching and accurate texture drawing based on the UV data before and after occlusion, projecting the photo texture and disease annotations onto the DZI unfolded map; S5: Perform dual-view disease linkage visualization and export. The same disease annotation is displayed on the 3D model view through ShaderMaterial overlay and on the DZI unfolded view through SVG layer overlay. Finally, the processed disease data is exported in multiple standardized formats.

2. The method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, In step S1, the aerial triangulation process is as follows: the multi-view real-scene photos are reconstructed into three dimensions using the motion recovery structure algorithm to obtain a GLB three-dimensional model for three-dimensional display and ray occlusion detection and a PLY three-dimensional model for DZI unfolded coordinate mapping.

3. The method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, In step S2, creating the view cone includes: calculating the field of view (FOV) based on the intrinsic parameters obtained from the analysis, and filtering and screening disabled cameras and uncalibrated cameras; calculating the position and orientation of the camera in the world coordinate system based on the extrinsic parameters, and defining the spatial range of the view cone.

4. The method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, Step S3 includes: sequentially performing backface culling, perspective projection, distortion correction, UV normalization, UV boundary verification, and ray occlusion detection on the vertices of the GLB 3D model, filtering out valid projected UV coordinates and valid marker positions, and performing texture and hazard annotation bonding rendering only on valid vertices.

5. The method for three-dimensional infrastructure image projection and spatial location of defects according to claim 4, characterized in that, The distortion correction is a full-parameter forward coupling correction method that integrates radial distortion, tangential distortion, and affine distortion. It performs normalization deviation compensation and denormalization restoration on the projected pixel coordinates, eliminates the projection offset caused by lens distortion, and obtains high-precision distortion-free projection coordinates.

6. A method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, In step S4, establishing the bidirectional coordinate mapping relationship between the PLY raster and the DZI unfolded image includes: based on the row and column index of the PLY raster unit, the overall size of the raster, and the resolution parameters of the DZI unfolded image, the conversion of the three-dimensional PLY raster coordinates to the two-dimensional DZI unfolded image pixel coordinates is completed through linear mapping, and a bidirectional queryable coordinate correspondence relationship is constructed.

7. A method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, Step S4 includes: projecting the photo texture onto the DZI unfolded image through affine transformation based on the UV after occlusion; performing AABB overlap detection on the UV range of the disease and PLY grid cells based on the UV before occlusion; establishing the correspondence between the disease and unfolded image cells; and projecting the disease label onto the DZI unfolded image.

8. The method for three-dimensional infrastructure image projection and spatial location of defects according to claim 7, characterized in that, The method of projecting the photo texture onto the DZI unfolded image by means of occlusion and UV mapping includes: calculating a three-point affine transformation matrix for the four effective UVs at the four corners of each PLY grid cell, and then performing Canvas / WebGL rasterization on the PLY raster texture area after affine transformation correction, thereby projecting the local texture of the photo onto the DZI unfolded image.

9. A method for three-dimensional infrastructure image projection and spatial location of defects according to claim 1, characterized in that, Step S5 includes: automatically deduplicating, performing connectivity analysis and OBB bounding box calculation on all disease annotations on the DZI unfolded map, and exporting the optimized disease data as interactive SVG, structured CSV and 3D disease marker JSON files.

10. A system for three-dimensional infrastructure image projection and spatial localization of defects, characterized in that, The system includes: The data acquisition and preprocessing module, configured in the data acquisition and preprocessing layer, is used to acquire real-scene photos, reconstruct GLB and PLY 3D models through aerial triangulation, solve camera parameter files, and obtain JSON format disease annotation data. The projection calculation and mapping module, configured in the projection calculation and mapping layer, is used to complete camera parameter parsing, conversion between the aerial coordinate system and the Three.js standard coordinate system, creation of camera view cone and construction of virtual camera, and to perform projection calculation, distortion correction and occlusion detection on the vertices of the GLB model, generating the effective projected UV coordinates and effective flags of each vertex. The coordinate mapping and unfolding module, configured in the coordinate mapping and unfolding layer, is used to construct the bidirectional coordinate mapping relationship between the PLY grid and the DZI unfolded image, solve the three-point affine transformation matrix of the PLY grid cell, complete the AABB matching and binding of the disease and the grid cell, and establish the correspondence between the disease and the unfolded image cell. The visualization and export module, configured in the visualization and export layer, includes a dual-view visualization unit and a multi-format export unit. It is used to realize the linkage display of disease in dual views of 3D view and DZI unfolded diagram, optimize the disease data, and finally export the results as interactive SVG, structured CSV, and 3D disease marker JSON standardized output files.

Images