A method, system, apparatus, and medium for quantifying areas of building structure damage

Abstract

The application provides a method, system, device and medium for quantifying a damaged area of a building structure, which comprises the following steps: collecting initial panoramic images and corresponding laser radar point cloud data before and after damage; correcting fish-eye distortion by using geometric consistency of straight lines of the structure as a constraint, overcoming the fish-eye distortion of the images, and obtaining optimized panoramic images before and after damage with accurate texture references; registering a plurality of optimized panoramic images of a target area to be detected after damage and an optimized panoramic image before damage corresponding to the target area, and obtaining an optimized registered panoramic image; fusing the optimized registered panoramic image and corresponding laser radar point cloud data, and obtaining a high-precision integrated fusion result of texture and geometric information; and finally obtaining a segmentation result of the target area after damage by using a preset multi-scale target detection model, combining the segmentation result and the fusion result, and obtaining an accurate damage quantification result of the target area after damage.

Description

Technical Field

[0001] This invention belongs to the field of building structure testing technology, specifically relating to a method, system, equipment, and medium for quantitatively measuring the damaged area of ​​a building structure. Background Technology

[0002] Panoramic cameras and LiDAR, as crucial sensing devices for building structure inspection and spatial measurement, have been widely applied in recent years in fields such as building monitoring, post-disaster assessment, and structural safety diagnosis. Panoramic cameras provide high-resolution global texture information, while LiDAR provides high-precision spatial depth information. Theoretically, their combination can achieve high-precision 3D reconstruction and quantitative analysis of building damage. However, in practical applications, existing fusion measurement methods based on single devices or dual sensors still have a series of technical limitations, making it difficult to meet the comprehensive needs for full-area coverage, accurate measurement, and automated analysis in complex environments.

[0003] Panoramic cameras, with their large field of view and high-resolution imaging capabilities, are widely used for building facade inspection, crack identification, and structural change analysis. However, existing panoramic image-based detection methods suffer from the following problems: 1. Severe fisheye distortion: Panoramic images are typically acquired using fisheye lenses, resulting in significant radial and tangential distortion. Traditional correction models based on fixed intrinsic parameters (such as pinhole models or polynomial models) struggle to adequately eliminate nonlinear distortion, leading to distorted linear structures and scale inaccuracies, thus affecting subsequent geometric measurements and registration accuracy. 2. Most image distortion correction algorithms rely solely on geometric models or single-target calibration, neglecting prior structural information about the scene itself. The correction results often fail to meet the geometric accuracy requirements of building measurements. 3. Panoramic cameras can only acquire two-dimensional texture images, unable to directly reflect surface geometric deformation or structural depressions, limiting the quantitative analysis of parameters such as the actual area of ​​peeling.

[0004] LiDAR, by actively emitting laser pulses and receiving echo signals, can obtain high-precision 3D point cloud data and has been widely used in building volume mapping, deformation monitoring, and safety assessment. However, the application of LiDAR in building damage detection still has the following limitations: 1. LiDAR reflection signals only contain geometric coordinates and reflection intensity, which cannot distinguish differences in surface materials and is difficult to identify subtle texture features such as cracks, peeling, and stains, resulting in ambiguous positioning of damaged areas. 2. In complex building environments, LiDAR sampling is affected by reflection angle and surface roughness, easily resulting in data gaps and shadow areas, especially in facades and corners where the point cloud is sparse and cannot fully reflect the surface morphology of the structure. 3. The point cloud data volume is huge, and traditional ICP (Iterative Closest Point) based registration algorithms have slow convergence speed and are prone to getting trapped in local optima, making it difficult to achieve fast and high-precision alignment in large-scale scenes.

[0005] Currently, scholars both domestically and internationally have begun to explore combining panoramic cameras with LiDAR to achieve integrated texture-geometry measurement of architectural scenes. However, the following problems still exist: 1. Panoramic images are spherical projections, while point clouds are sampled using Cartesian coordinates. The two coordinate systems differ significantly. Without distortion correction and unified projection, misalignment, drift, or "wall-penetrating" artifacts are prone to occur after fusion. 2. The fusion process relies on manual intervention, making it impossible to achieve a fully automated closed loop of "acquisition-fusion-analysis".

[0006] Therefore, this application anticipates a method for quantitative analysis of building structural damage based on the fusion of panoramic camera and lidar. Summary of the Invention

[0007] To address the technical problems in existing technologies for measuring building structural damage, such as high risks associated with manual surveying, difficulty in eliminating fisheye distortion, low accuracy in image registration before and after damage, reliance on manual operation for point cloud-image fusion, and lack of unified quantitative evaluation standards, this invention provides a method, system, device, and medium for quantifying the damaged area of ​​a building structure.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] In a first aspect, embodiments of this disclosure provide a method for quantifying the area of ​​structural damage in a building, comprising the following steps:

[0010] Step S1: Collect initial panoramic images and corresponding LiDAR point cloud data of the building structure before damage, as well as multiple initial panoramic images and corresponding LiDAR point cloud data of the target area after damage.

[0011] Step S2: Using fisheye distortion correction constrained by the geometric consistency of the structural straight lines, the initial panoramic images before and after the damage are corrected to obtain optimized panoramic images before and after the damage.

[0012] Step S3: Select multiple optimized panoramic images after destruction and one optimized panoramic image before destruction corresponding to the target area, and register them to obtain an optimized registered panoramic image;

[0013] Step S4: Based on the optimized registration panoramic image and the corresponding lidar point cloud data, project the lidar point cloud data onto the plane of the optimized registration panoramic image to obtain the fusion result;

[0014] Step S5: Based on the preset multi-scale target detection model, obtain the segmentation result of the damaged target region, and combine the segmentation result with the fusion result to obtain the quantitative result of the damage to the damaged target region.

[0015] Furthermore, the fisheye distortion correction in step S2 includes the following steps:

[0016] Step A1: Offline calibration of the panoramic camera is performed using the checkerboard calibration method. Multiple sets of checkerboard images are acquired under various poses and viewpoints, corner features of the checkerboard images are extracted, and the intrinsic parameter matrix of the panoramic camera is solved using nonlinear least squares optimization. With distortion coefficient matrix ;

[0017] ;

[0018] ;

[0019] In the formula, This indicates the focal length of the panoramic camera along the x-axis. This represents the offset of the origin of the image coordinate system along the x-axis relative to the center of the imaging plane; This represents the focal length of the panoramic camera along the y-axis. This represents the offset of the origin of the image coordinate system along the y-axis relative to the center of the imaging plane; , and Both represent radial distortion coefficients; and Both represent tangential distortion coefficients;

[0020] Step A2: Based on the intrinsic parameter matrix and distortion coefficient matrix Perform distortion correction on the initial panoramic image to obtain the initial corrected image. ;

[0021] Step A3: Extract the initial corrected image using the Canny edge detection operator. The significant edge information is obtained, and the initial corrected image is detected by probabilistic Hough transform. The set of line segments in the structure, along with significant edge information, serves as a reference set for line constraints, forming constraints for the geometric consistency of the structural lines.

[0022] Step A4: Extract each structural straight line from the linear constraint reference set. pixel set The vertical distance from each pixel to its corresponding fitted line is calculated as the local geometric error. With minimizing the curvature of the line as the optimization objective, the local geometric errors of the line constraint reference set are fitted using the least squares method to construct a global line curvature error function. And based on the global straight line bending error function The distortion coefficient matrix D is optimized; based on the optimized distortion coefficient matrix D and the intrinsic parameter matrix... For the initial corrected image Perform secondary distortion correction;

[0023] ;

[0024] In the formula, Represented by the distortion coefficient matrix Controlled anti-distortion mapping function; An ideal linear model fitted by the least squares method;

[0025] Step A5: When the residual of the straight line error fitting is less than the preset pixel threshold and the residual distortion rate is less than the preset distortion rate threshold, output the optimized panoramic image before and after the destruction.

[0026] Furthermore, when obtaining the optimized registered panoramic image in step S3, the following steps are included:

[0027] Step B1: Scale-Invariant Feature Transform (SIFT) algorithm is used to extract key points and corresponding feature descriptors from the optimized panoramic images before and after the damage. Nearest neighbor search is then performed using the Euclidean distance of the feature descriptors to establish an initial feature matching set between the optimized panoramic images before and after the damage. :

[0028] ;

[0029] In the formula, Key points of the optimized panoramic image before damage; Key points of the optimized panoramic image after destruction; Represents the initial feature matching set The total number of matching point pairs in the middle; Represents the initial feature matching set The index of the i-th matching point pair is a positive integer from 1 to N;

[0030] Step B2: Based on the initial feature matching set Extract the corresponding optimized panoramic image before the damage. And the optimized panoramic image after destruction The significant linear features are identified, and sets of linear lines are constructed for the optimized panoramic images before and after the destruction, including:

[0031] ;

[0032] ;

[0033] In the formula, Indicate quantity;

[0034] Step B3: Based on the initial feature matching set Each initial matching point pair Based on the consistency of distance and direction to the nearest structural line in the set of lines, initial matching point pairs with higher consistency are selected. Construct an optimized matching set , ;

[0035] Step B4: Based on optimizing the matching set The keypoint sets of the optimized panoramic image before and after damage are extracted. The homography matrix between the optimized panoramic image before and after damage is estimated using the Random Sample Consensus Algorithm (RANSAC). Based on homography matrix The overall geometric transformation relationship between the optimized panoramic images before and after the destruction is obtained under the assumption of planar projection, and the initial registered panoramic image is obtained based on the overall geometric transformation relationship;

[0036] Step B5: Calculate the initial matching point pairs The initial matching error and the root mean square error of the initial registered panoramic image are used to determine the actual improvement rate of the root mean square error of the registration. If the actual improvement rate is greater than or equal to the preset improvement rate, the initial registered panoramic image is used as the optimized registered panoramic image; otherwise, the parameters of the Random Sample Consensus Algorithm (RANSAC) are adjusted, and the homography matrix is ​​re-estimated. The system then regenerates the initial registered panoramic image, judges the actual improvement rate, and continues until the actual improvement rate meets the preset requirements or reaches the maximum number of iterations. Finally, it outputs the current optimal registered panoramic image as the optimized registered panoramic image.

[0037] Furthermore, when selecting initial matching points with high consistency, if the initial matching points are... If each initial matching point falls within the neighborhood of a salient linear feature in the optimized panoramic image before and after the destruction, and satisfies the consistency constraint relative to the direction of the corresponding salient linear feature, then the initial matching point pair is determined to be... If a match is valid, it is considered a false match and discarded, based on a valid initial match pair. Construct the optimized matching set .

[0038] Furthermore, obtaining the fused depth map in step S4 includes the following steps:

[0039] Step C1: Perform joint calibration of multiple sensors based on the checkerboard calibration method to obtain the extrinsic parameter relationship between the LiDAR and the panoramic camera;

[0040] Step C2: Based on the aforementioned extrinsic parameter relationship, perform coordinate transformation on the lidar point cloud data to obtain the point cloud projection coordinates in the panoramic camera coordinate system. :

[0041] ;

[0042] In the formula, The homogeneous coordinates of the lidar point cloud data. It is a complete set of lidar point cloud data. a subset of , , and They are respectively Three-dimensional coordinate components in the lidar coordinate system The coordinates are in the horizontal direction. The coordinates are in the vertical direction. The coordinates are in the depth direction. It is a translation vector; It is a rotation matrix; This is the intrinsic parameter matrix;

[0043] Step C3: Based on the intrinsic parameter matrix K and the panoramic camera fisheye projection model, project the point cloud coordinates. Projecting onto the plane of the optimized registered panoramic image generates visible points. During projection, only the point cloud projection coordinates located in front of the panoramic camera and whose projection coordinates fall within the effective field of view of the optimized registered panoramic image are retained. If multiple point cloud projection coordinates are projected onto the same pixel or adjacent pixel area, the Z-buffer depth caching strategy is used to retain only the point cloud projection coordinates closest to the panoramic camera as visible points.

[0044] Step C4: Based on the projection position of each visible point in the optimized registration panoramic image, extract the color information of the corresponding pixel in the optimized registration panoramic image, map it to the visible point, generate color point cloud data, and align the visible points with the optimized registration panoramic image at the pixel level. Project the point cloud coordinates of the visible points. The Z-axis coordinates are used as projection depth information. The depth information is organized into dense or semi-dense depth maps according to pixel positions. The color point cloud data and the depth map are the fusion result.

[0045] Furthermore, when obtaining the segmentation result of the damaged target region in step S5, the following steps are included:

[0046] Step D1: Input the optimized panoramic image into the multi-scale object detection model. The multi-scale object detection model extracts feature representations at different levels through layer-by-layer convolution and downsampling operations to obtain multi-scale feature maps from local texture to high-level semantics.

[0047] Step D2: Based on the multi-scale context fusion structure, the features under different receptive fields of the multi-scale feature map are encoded and fused in parallel to obtain the candidate detection box and the corresponding confidence of the damaged target region;

[0048] Step D3: Mark the inner region of the candidate detection box as the potential foreground and the outer region of the candidate detection box as the background to construct the initial mask of the target region after destruction;

[0049] Step D4: Establish Gaussian mixture models for the pixel color distribution of the initial mask for the foreground and background respectively, and update the segmentation results of the foreground and background under the improved graph cut optimization framework, taking into account the spatial continuity constraints between pixels. When updating the segmentation results of the foreground and background under the improved graph cut optimization framework, the following steps are performed:

[0050] Step D41: Shrink the initial mask boundary corresponding to the candidate detection box inward by a preset number of pixels to optimize the boundary of the initial mask and generate a binary mask;

[0051] Step D42: Perform connected component analysis on the binary mask, traverse all foreground connected components, and retain only the connected component with the largest area to generate the foreground mask;

[0052] Step D43: Perform morphological closing operation on the foreground mask using a 5×5 elliptical structuring element. During the closing operation, use a dilation-then-erosion sequence to fill the tiny holes inside the foreground region until the preset iteration threshold is reached to obtain the segmentation result.

[0053] Furthermore, in obtaining quantitative results of destruction, the following steps are included:

[0054] Step E1: Based on the segmentation results, count the number of pixels in the damaged area in the image coordinate system, and the set of pixel coordinates of the damaged area in the image. Use the number of pixels as the pixel area of ​​the damaged area, and the set of pixel coordinates as the pixel spatial distribution position of the damaged area.

[0055] Step E2: Based on the projection relationship between the LiDAR point cloud and the panoramic camera image in the fusion result, and combined with the pixel area and pixel spatial distribution of the damaged area, the pixel coordinates corresponding to the damaged area are mapped to the three-dimensional space under the panoramic camera coordinate system to obtain the real spatial scale information of the damaged area.

[0056] Step E3: Extract the point cloud depth data corresponding to the spatial location of pixels in the damaged area, calculate the actual physical scale corresponding to a single pixel, and multiply the pixel scale by the pixel area of ​​the damaged area to obtain the three-dimensional actual area of ​​the damaged area.

[0057] The spatial distribution and three-dimensional actual area of ​​the damaged area are used as quantitative results of the damage.

[0058] In a second aspect, embodiments of this disclosure provide a system for quantifying the area of ​​structural damage in a building, comprising:

[0059] The acquisition unit is configured to acquire initial panoramic images and corresponding lidar point cloud data of the building structure before it is damaged, as well as multiple initial panoramic images and corresponding lidar point cloud data of the target area after the damage.

[0060] The correction unit is configured to use fisheye distortion correction constrained by the geometric consistency of the structural straight lines to correct the initial panoramic images before and after the damage, thereby obtaining optimized panoramic images before and after the damage.

[0061] The registration unit is configured to select multiple optimized panoramic images after destruction and an optimized panoramic image before destruction corresponding to the target area, and perform registration to obtain an optimized registered panoramic image.

[0062] The fusion unit is configured to project the lidar point cloud data onto the plane of the optimized registration panoramic image based on the optimized registration panoramic image and the corresponding lidar point cloud data to obtain the fusion result;

[0063] The output unit is configured to obtain the segmentation result of the damaged target region based on a preset multi-scale target detection model, and to obtain the quantitative result of the damage to the damaged target region by combining the segmentation result with the fusion result.

[0064] In a third aspect, embodiments of this disclosure provide an electronic device, characterized in that the electronic device comprises:

[0065] At least one processor; and,

[0066] The memory is communicatively connected to the at least one processor; wherein,

[0067] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for quantifying the area of ​​structural damage.

[0068] In a fourth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method for quantifying the area of ​​structural damage.

[0069] Compared with the prior art, the present invention has the following beneficial technical effects:

[0070] This invention provides a method, system, device, and medium for quantifying the damaged area of ​​a building structure. The method acquires initial panoramic images before and after damage, along with corresponding LiDAR point cloud data. It then uses fisheye distortion correction constrained by the geometric consistency of the structural straight lines to overcome image fisheye distortion, resulting in optimized panoramic images with accurate texture references before and after damage. Multiple optimized panoramic images of the target area to be detected after damage, along with an optimized panoramic image of the target area before damage, are registered. The optimized panoramic image that best matches the target area before damage is found, resulting in an optimized registered panoramic image. This optimized registered panoramic image is then fused with the corresponding LiDAR point cloud data to obtain a high-precision integrated fusion result of texture and geometric information. Finally, a pre-set multi-scale target detection model is used to obtain the segmentation result of the target area after damage. Combining the segmentation result and the fusion result, an accurate quantitative result of the damage to the target area after damage is obtained. This method eliminates the manual point selection and fusion steps, achieving fully automated processing from "acquisition—fusion—detection—evaluation," significantly improving registration and detection accuracy. It enables precise measurement and visual evaluation of building damage, providing a high-precision, automated, and scalable technical solution for building safety damage detection. Attached Figure Description

[0071] Figure 1 A flowchart illustrating a method for quantifying the damage zone of a building structure according to an embodiment of this disclosure is shown.

[0072] Figure 2 A flowchart of a method for correcting fisheye distortion according to an embodiment of this disclosure is shown;

[0073] Figure 3 A flowchart illustrating a registration method according to an embodiment of this disclosure is shown.

[0074] Figure 4 A flowchart illustrating the method for obtaining the fusion result according to an embodiment of this disclosure is shown;

[0075] Figure 5 A flowchart illustrating a method for obtaining quantitative results of destruction according to an embodiment of this disclosure is shown;

[0076] Figure 6 The accompanying diagram shows a comparison before and after fisheye distortion correction in Scenario 1 of the present disclosure.

[0077] Figure 7 The accompanying diagram shows a comparison before and after fisheye distortion correction in Scenario 2 of the present disclosure.

[0078] Figure 8 An optimized panoramic image of the target area before destruction is shown in an embodiment of this disclosure;

[0079] Figure 9An optimized panoramic image of the target area after destruction, according to an embodiment of this disclosure, is shown.

[0080] Figure 10 The image extracted using the scale-invariant feature transform algorithm according to an embodiment of this disclosure is shown;

[0081] Figure 11 An optimized registered panoramic image according to an embodiment of this disclosure is shown;

[0082] Figure 12 This illustration shows a lidar point cloud data map corresponding to the optimized registration panoramic image according to an embodiment of the present disclosure;

[0083] Figure 13 A color point cloud data diagram of an embodiment of this disclosure is shown;

[0084] Figure 14 A depth map of an embodiment of this disclosure is shown;

[0085] Figure 15 This diagram illustrates the process of obtaining the segmentation result in scenario three of the embodiments of this disclosure.

[0086] Figure 16 This diagram illustrates the process of obtaining the segmentation result in scenario four of the present disclosure.

[0087] Figure 17 A diagram of an apparatus for quantifying the area of ​​structural damage in a building, according to an embodiment of this disclosure, is shown. Detailed Implementation

[0088] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0089] Figure 1 A flowchart 100 of a method for quantifying the damage zone of a building structure according to an embodiment of the present disclosure is shown. The embodiment of the present disclosure includes the following steps:

[0090] In step S101, an initial panoramic image and corresponding lidar point cloud data are acquired before the building structure is damaged, as well as multiple initial panoramic images and corresponding lidar point cloud data of the target area after the damage.

[0091] Specifically, when acquiring the initial panoramic image of the target area after damage, it is impossible to guarantee that the acquisition positions before and after damage are the same. Therefore, it is necessary to select the corresponding initial panoramic image before damage based on the target area to be detected after damage, and to acquire multiple images based on the possible acquisition positions of the corresponding initial panoramic image before damage, so as to obtain multiple initial panoramic images and corresponding LiDAR point cloud data.

[0092] Next, proceed to step S102.

[0093] In step S102, fisheye distortion correction with structural straight-line geometric consistency as constraint is used to correct the initial panoramic images before and after the damage, thereby obtaining optimized panoramic images before and after the damage.

[0094] To address the severe radial distortion caused by the ultra-wide field of view introduced by panoramic cameras in architectural scene imaging, this embodiment employs an adaptive fisheye distortion correction scheme, constrained by the geometric consistency of the structural straight lines. During fisheye distortion correction, an intrinsic parameter matrix is ​​generated using a checkerboard calibration method based on the calibration model. With distortion coefficient matrix And by using the straight line features in real architectural scenes to analyze the distortion coefficient matrix Iterative optimization is performed to improve the geometric accuracy of the correction results in real-world engineering environments. Figure 2 A flowchart 200 illustrating a method for correcting fisheye distortion according to an embodiment of this disclosure is shown, as follows: Figure 2 As shown, the fisheye distortion correction in step S102 includes the following steps:

[0095] Step A1: Offline calibration of the panoramic camera is performed using the checkerboard calibration method. Multiple sets of checkerboard images are acquired under various poses and viewpoints, corner features of the checkerboard images are extracted, and the intrinsic parameter matrix of the panoramic camera is solved using nonlinear least squares optimization. With distortion coefficient matrix ;

[0096] ;

[0097] ;

[0098] In the formula, This indicates the focal length of the panoramic camera along the x-axis. This represents the offset of the origin of the image coordinate system along the x-axis relative to the center of the imaging plane; This represents the focal length of the panoramic camera along the y-axis. This represents the offset of the origin of the image coordinate system along the y-axis relative to the center of the imaging plane; , and Both represent radial distortion coefficients; and Both represent tangential distortion coefficients;

[0099] Step A2: Based on the intrinsic parameter matrix and distortion coefficient matrix Perform distortion correction on the initial panoramic image to obtain the initial corrected image. ;

[0100] Step A3: Extract the initial corrected image using the Canny edge detection operator. The significant edge information is obtained, and the initial corrected image is detected by probabilistic Hough transform. The set of line segments in the model is used as a reference set for line constraints, combining significant edge information with the set of line segments to form constraints on the geometric consistency of structural lines. Specifically, the reference set for line constraints essentially consists of structural line features such as building walls and beams / columns. Combined with prior geometric knowledge of the building, the detected line segments are filtered. The prior geometric knowledge of the building refers to the inherent geometric characteristics of the building structure, such as building walls being mostly horizontal or vertical, beam and column outlines being mostly straight with fixed angles (e.g., 0° / 90°), and the length of structural lines being much greater than that of non-structural noise line segments (e.g., decorative lines, cracks). After line detection, line segments with angles of 0°±5° or 90°±5° (conforming to the horizontal / vertical structural characteristics of the building) are first filtered out. Then, line segments are filtered based on the following quantification rules, ultimately constructing the reference set for line constraints. These quantification rules include:

[0101] Longer length: The pixel length of a straight line segment is greater than or equal to a preset threshold (e.g., 200 pixels, which can be adjusted according to the image resolution). Line segments shorter than this threshold are judged as short line segments and are discarded.

[0102] Good continuity: The pixel breakpoint rate of the straight line segment is ≤5% (breakpoint rate = number of breakpoint pixels / total number of pixels in the straight line segment). Line segments with a breakpoint rate higher than 5% are judged as discontinuous and discarded.

[0103] In addition, priority is given to retaining straight lines that meet the criteria of "longer length + better continuity + conformity to architectural geometry a priori", specifically structural straight lines such as building wall edges and beam and column outlines, while non-structural straight lines such as decorative lines, cracks, and noise are eliminated.

[0104] Step A4: Extract each structural straight line from the linear constraint reference set. pixel set The vertical distance from each pixel to its corresponding fitted line is calculated as the local geometric error. With minimizing the curvature of the line as the optimization objective, the local geometric errors of the line constraint reference set are fitted using the least squares method to construct a global line curvature error function. And based on the global straight line bending error function The distortion coefficient matrix D is optimized; based on the optimized distortion coefficient matrix D and the intrinsic parameter matrix... For the initial corrected image Perform secondary distortion correction;

[0105] ;

[0106] In the formula, Represented by the distortion coefficient matrix Controlled anti-distortion mapping function; An ideal linear model fitted by the least squares method;

[0107] Step A5: When the residual of the straight-line error fitting is less than a preset pixel threshold and the residual distortion rate is less than a preset distortion rate threshold, output the optimized panoramic image before and after the destruction. Specifically, in this embodiment, the preset pixel threshold is 0.2 × 10⁻³, and the preset distortion rate threshold is 13%. When both the preset pixel threshold condition and the preset distortion rate threshold are satisfied, the distortion coefficient matrix is ​​considered to be... The system has reached a stable convergence state and outputs an optimized panoramic image. This optimized panoramic image significantly outperforms traditional calibration results in terms of geometric consistency and structural straightness preservation, providing a high-precision geometric benchmark for subsequent image registration, multi-source data fusion, and building structure measurement. Figure 6 The accompanying images show comparisons of fisheye distortion correction before and after scenarios one and two in embodiments of this disclosure. Figure 6 and Figure 7 The images on the left and right are the initial panoramic images before fisheye distortion correction, while the images on the right are the optimized panoramic images after fisheye distortion correction.

[0108] Next, proceed to step S103.

[0109] In step S103, multiple optimized panoramic images after destruction and an optimized panoramic image before destruction corresponding to the target area are selected and registered to obtain an optimized registered panoramic image.

[0110] To address the registration difficulties in panoramic images before and after building damage under conditions of changes in shooting posture, viewpoint, and local structure, this embodiment first utilizes scale- and rotation-invariant local features to establish an initial feature matching set. As an initial correspondence, and based on this, a set of straight lines that are stable in the architectural scene is introduced to form a set of straight lines, which serves as a geometric constraint to eliminate mismatches in the initial correspondence. During registration, weighted optimization is performed, thereby improving the registration accuracy of the optimized panoramic images before and after damage under complex structural changes. Specifically, Figure 3 A flowchart 300 illustrating a registration method according to an embodiment of the present disclosure is shown. Step S103, when obtaining the optimized registered panoramic image, includes the following steps:

[0111] Step B1: Scale-Invariant Feature Transform (SIFT) algorithm is used to extract key points and corresponding feature descriptors from the optimized panoramic images before and after the damage. Nearest neighbor search is then performed using the Euclidean distance of the feature descriptors to establish an initial feature matching set between the optimized panoramic images before and after the damage. :

[0112] ;

[0113] In the formula, Key points of the optimized panoramic image before damage; Key points of the optimized panoramic image after destruction; Represents the initial feature matching set The total number of matching point pairs in the middle; Represents the initial feature matching set The index of the i-th matching point pair is a positive integer from 1 to N;

[0114] It should be noted that, through the scale-space pyramid, keypoints with significant local extrema in image features are detected, and rotation-invariant feature descriptors are generated for each keypoint. Significant local extrema refer to extreme points in different layers and directional neighborhoods of the scale-space pyramid where the response value is significantly greater than that of surrounding pixels. This is the core criterion for the Scale Invariant Feature Transform (SIFT) algorithm to select stable keypoints.

[0115] Step B2: Based on the initial feature matching set Extract the corresponding optimized panoramic image before the damage. And the optimized panoramic image after destruction The significant straight line features are derived from the structural elements of buildings such as walls, beams, columns, and door / window edges in the image. Sets of straight lines are constructed from the optimized panoramic images before and after the destruction, including:

[0116] ;

[0117] ;

[0118] In the formula, Indicate quantity;

[0119] Step B3: Based on the initial feature matching set Each initial matching point pair Based on the consistency of distance and direction to the nearest structural line in the set of lines, initial matching point pairs with higher consistency are selected. Construct an optimized matching set , This is used to eliminate mismatched point pairs that deviate from the main geometric direction of the building;

[0120] Step B4: Based on optimizing the matching set The keypoint sets of the optimized panoramic image before and after damage are extracted. The homography matrix between the optimized panoramic image before and after damage is estimated using the Random Sample Consensus Algorithm (RANSAC). Based on the homography matrix, the overall geometric transformation relationship between the optimized panoramic images before and after the destruction is obtained under the plane projection assumption. Based on the overall geometric transformation relationship, the initial registered panoramic image is obtained.

[0121] Step B5: Calculate the initial matching point pairs The initial matching error and the root mean square error (RMSE) of the initial registered panoramic image are used to determine the actual improvement rate of the RMSE. If the actual improvement rate is greater than or equal to a preset improvement rate, the initial registered panoramic image is used as the optimized registered panoramic image. Otherwise, the parameters of the Random Sample Consensus Algorithm (RANSAC), including the number of iterations and the inlier threshold, are adjusted. The homography matrix is ​​re-estimated, and a new initial registered panoramic image is generated. The actual improvement rate is then judged again until the actual improvement rate meets the preset requirement or reaches the maximum number of iterations. Finally, the current optimal registered panoramic image is output as the optimized registered panoramic image. Specifically, in this embodiment, the preset improvement rate is 55%.

[0122] Furthermore, when selecting initial matching points with high consistency, if the initial matching points are... If each initial matching point falls within the neighborhood of a salient linear feature in the optimized panoramic image before and after the destruction, and satisfies the consistency constraint relative to the direction of the corresponding salient linear feature, then the initial matching point pair is determined to be... If a match is valid, it is considered a false match and discarded, based on a valid initial match pair. Construct the optimized matching set .

[0123] In this embodiment, Figure 8 and Figure 9 The optimized panoramic image before and after destruction corresponding to the target area in this embodiment of the present disclosure are shown respectively. Figure 10 The image shows the registration result after direct feature matching and homography transformation using the existing Scale Invariant Feature Transform (SIFT) algorithm, without introducing linear constraints from the building structure. Figure 11 The image shown is the final optimized registered panoramic image obtained using the SIFT registration algorithm with linear constraints of building structures proposed in this embodiment. The registration effect for the target area in this embodiment includes two sets of verification data, which are described in detail below:

[0124] To clearly compare the effects of the two registration methods, two sets of quantitative evaluation results are presented:

[0125] In existing related technologies, the root mean square error (RMSE) of the matching points before registration is 1198.7856 pixels; after registration, the RMSE is 459.6952 pixels, representing an RMSE improvement of 61.65%. This improvement rate is higher than the preset improvement rate of 55%, meeting the basic registration requirements. The SIFT registration algorithm proposed in this embodiment, which integrates linear constraints on the building structure, yields a final optimized registered panoramic image. The initial number of SIFT matching point pairs is 402 pairs, and the number of effective matching point pairs after filtering by linear constraints on the building structure is 240 pairs. The RMSE before registration after linear constraints is 1234.0750 pixels, and the RMSE after registration is 0.4841 pixels, representing an RMSE improvement of 99.96%. It can be seen that the improvement rate of the SIFT registration algorithm proposed in this embodiment is much higher than 55.0%, and the RMSE after registration is ≤1.5 pixels, meeting the stringent requirements for high-precision registration in building structure damage measurement.

[0126] Next, proceed to step S104.

[0127] In step S104, based on the optimized registration panoramic image and the corresponding lidar point cloud data, the lidar point cloud data is projected onto the plane of the optimized registration panoramic image to obtain the fusion result.

[0128] In this embodiment, in order to achieve the coordinated expression of high-precision geometric information of LiDAR point cloud data and high-resolution texture information of panoramic image in architectural scenes, the three-dimensional point cloud is accurately projected onto the two-dimensional panoramic image plane by unifying the spatial coordinate system of LiDAR and panoramic camera, thereby realizing the integrated fusion expression of geometry and texture and obtaining the fusion result. Figure 4 A flowchart 300 illustrating a method for obtaining a fusion result according to an embodiment of this disclosure is shown. Step S104, obtaining the fusion result, includes the following steps:

[0129] Step C1: Perform joint calibration of multiple sensors based on the checkerboard calibration method to obtain the extrinsic parameter relationship between the LiDAR and the panoramic camera; specifically, the joint calibration of multiple sensors based on the checkerboard calibration method includes:

[0130] Offline calibration of the panoramic camera was performed using a checkerboard calibration method. Multiple sets of checkerboard images were acquired under various poses and viewpoints, corner features of the checkerboard images were extracted, and the intrinsic parameter matrix of the panoramic camera was solved using nonlinear least squares optimization. With distortion coefficient matrix ;

[0131] ;

[0132] ;

[0133] In the formula, This indicates the focal length of the panoramic camera along the x-axis. This represents the offset of the origin of the image coordinate system along the x-axis relative to the center of the imaging plane; This represents the focal length of the panoramic camera along the y-axis. This represents the offset of the origin of the image coordinate system along the y-axis relative to the center of the imaging plane; , and Both represent radial distortion coefficients; and Both represent tangential distortion coefficients.

[0134] When obtaining the extrinsic parameter relationship, a checkerboard calibration board is deployed within the common field of view of the LiDAR and the panoramic camera. Corner features of the checkerboard in the optimized registration panoramic image, as well as planar features in the LiDAR point cloud data, are extracted simultaneously. The corner features provide high-precision two-dimensional image positioning information, construct geometric constraints in the image coordinate system, and serve as a benchmark for calculating reprojection errors during extrinsic parameter optimization. The planar features characterize the geometric structure information of the checkerboard in three-dimensional space. Spatial normal vectors and planar parameters are obtained through plane fitting, providing three-dimensional geometric constraints for establishing the spatial correspondence between the LiDAR coordinate system and the camera coordinate system.

[0135] For corner and planar features, a least-squares optimization method is used to solve for the spatial transformation parameters between the panoramic camera and the LiDAR, obtaining the rotation matrix R, which characterizes the coordinate system rotation relationship, and the translation vector T, which characterizes the coordinate system translation relationship. The extrinsic parameter relationship formed by the rotation matrix R and the translation vector T serves as the rigid transformation relationship between the LiDAR and the panoramic camera, used to achieve the coordinate transformation from the LiDAR coordinate system to the panoramic camera coordinate system.

[0136] Step C2: Based on the aforementioned extrinsic parameter relationship, perform coordinate transformation on the lidar point cloud data to obtain the point cloud projection coordinates in the panoramic camera coordinate system. :

[0137] ;

[0138] In the formula, The homogeneous coordinates of the lidar point cloud data. It is a complete set of lidar point cloud data. a subset of , , ,and They are respectively Three-dimensional coordinate components in the lidar coordinate system The coordinates are in the horizontal direction. The coordinates are in the vertical direction. The coordinates are in the depth direction. This is a translation vector used to characterize the translation relationship between the lidar coordinate system and the panoramic camera coordinate system; This is a rotation matrix used to characterize the rotational relationship between the lidar coordinate system and the panoramic camera coordinate system; This is the intrinsic parameter matrix of the panoramic camera, used to map three-dimensional points in the camera coordinate system to two-dimensional homogeneous pixel coordinates in the image plane;

[0139] Step C3: Based on the intrinsic parameter matrix K and the panoramic camera fisheye projection model, which is the inherent imaging model of the panoramic camera and is used to describe the mapping relationship from three-dimensional spatial points to two-dimensional image pixels, the point cloud coordinates are projected. Project the image onto the plane of the optimized and registered panoramic image to generate visible points. During projection, apply the following filtering rules:

[0140] Only the point cloud projection coordinates located in front of the panoramic camera are retained. The region where the Z-axis coordinate is greater than 0, and the projected coordinates fall within the effective field of view of the optimized registered panoramic image;

[0141] Furthermore, when multiple point cloud projection coordinates are projected onto the same pixel or adjacent pixel area, the Z-buffer depth caching strategy is used to retain only the point cloud projection coordinates closest to the panoramic camera as visible points.

[0142] Step C4: Based on the projection position of each visible point in the optimized registration panoramic image, extract the color information of the corresponding pixel in the optimized registration panoramic image, and map it to the visible point to generate color point cloud data, such as... Figure 12 and Figure 13 As shown, the visible points are aligned pixel-level with the optimized registered panoramic image, and the point cloud coordinates of the visible points are projected. The Z-axis coordinate is used as the projected depth information, which is the depth value in the panoramic camera coordinate system. This depth information is organized into dense or semi-dense depth maps according to pixel positions. The color point cloud data and... Figure 14 The depth map shown is the fusion result.

[0143] Next, proceed to step S105.

[0144] In step S105, based on the preset multi-scale target detection model, the segmentation result of the damaged target region is obtained. Combining the segmentation result with the fusion result, the quantitative result of the damage to the damaged target region is obtained.

[0145] In this embodiment, to address the problem of high non-uniformity in scale, shape, and texture distribution of the damaged area of ​​the building, which has a large spatial scale span and irregular shape, it is difficult to achieve stable detection with a single scale feature. Robust detection of damaged targets at different scales is completed on the optimized panoramic image to obtain candidate detection boxes. On this basis, the candidate detection boxes are further finely segmented at the pixel level, and combined with the fusion results, the geometric parameters of the damaged area are automatically calculated.

[0146] Figure 5 A flowchart 400 illustrating a method for obtaining quantitative results of damage to a target region after destruction, according to an embodiment of this disclosure, is shown. Figure 5 As shown, when obtaining the segmentation result of the damaged target region in step S105, the following steps are included:

[0147] Step D1: Input the optimized panoramic image into the multi-scale object detection model. The multi-scale object detection model extracts feature representations at different levels through layer-by-layer convolution and downsampling operations to obtain multi-scale feature maps from local texture to high-level semantics. Specifically, in terms of network structure design, the multi-scale object detection model selects a lightweight convolutional neural network as the backbone network to reduce computational complexity and meet engineering deployment requirements.

[0148] Step D2: Based on the multi-scale context fusion structure, the features under different receptive fields of the multi-scale feature map are encoded and fused in parallel to obtain the candidate detection box and the corresponding confidence of the target region after destruction; the multi-scale context fusion structure adopts the Spatial Pyramid Pooling (SPP) module, which can simultaneously take into account local destruction features such as small cracks and macro-destruction forms such as large-area peeling and collapse.

[0149] Step D3: Since the candidate detection box can only provide a rough location of the damaged area and it is difficult to accurately describe the true outline and boundary shape of the damaged area, the inner area of ​​the candidate detection box is marked as the potential foreground and the outer area of ​​the candidate detection box is marked as the background, thus constructing the initial mask of the damaged target area.

[0150] Step D4: Establish Gaussian Mixture Models (GMMs) for the pixel color distribution of the initial mask for the foreground and background respectively. Combined with spatial continuity constraints between pixels, update the foreground and background partitioning results under an improved graph cut optimization framework. This improved framework is based on an engineering improvement of the GrabCut algorithm. When updating the foreground and background partitioning results, the improved graph cut optimization framework performs the following steps:

[0151] Step D41: Shrink the initial mask boundary corresponding to the candidate detection box inward by a preset number of pixels to optimize the boundary of the initial mask and generate a binary mask; in this embodiment, shrink it inward by 5 pixels to reduce the interference of background noise at the edge of the detection box on the foreground segmentation and improve the accuracy of the initial prior.

[0152] Step D42: Perform connected component analysis on the binary mask, traverse all foreground connected regions, and retain only the connected region with the largest area to generate the foreground mask; in order to eliminate small noise regions and avoid fragmented missegmentation;

[0153] Step D43: Perform morphological closing operation on the foreground mask using a 5×5 elliptical structuring element. During the closing operation, use a dilation-then-erosion sequence to fill the tiny holes inside the foreground region until the preset iteration threshold is reached to obtain the segmentation result.

[0154] The improved graph cut optimization framework effectively utilizes the statistical differences in color, texture, and edge distribution of the damaged regions to finely characterize irregular damaged boundaries, significantly reducing background missegmentation and boundary blurring, ultimately obtaining a high-precision binary mask for the damaged regions, such as... Figure 15 and 16 As shown.

[0155] Furthermore, in obtaining quantitative results of destruction, the following steps are included:

[0156] Step E1: Based on the segmentation results, count the number of pixels in the damaged area in the image coordinate system, and the set of pixel coordinates of the damaged area in the image. Use the number of pixels as the pixel area of ​​the damaged area, and the set of pixel coordinates as the pixel spatial distribution position of the damaged area.

[0157] Step E2: Based on the projection relationship between the LiDAR point cloud and the panoramic camera image in the fusion result, the projection relationship is composed of the camera intrinsic parameter matrix K, the rotation matrix R of the LiDAR to the camera, and the translation vector T. Combining the pixel area and pixel spatial distribution position of the damaged area, the pixel coordinates corresponding to the damaged area are mapped to the three-dimensional space under the panoramic camera coordinate system to obtain the real spatial scale information of the damaged area. The real spatial scale information includes three-dimensional coordinates and depth values, etc.

[0158] Step E3: Extract the point cloud depth data corresponding to the spatial distribution of pixels in the damaged area. The point cloud depth data is the Z-axis depth value in the panoramic camera coordinate system. Calculate the actual physical scale corresponding to a single pixel, which includes pixel width and pixel height. Pixel width = average depth value / camera intrinsic focal length. Pixel height = average depth value / camera intrinsic focal length The pixel scale is then multiplied by the pixel area of ​​the damaged region to complete the measurement conversion from pixel scale to physical scale, thus obtaining the actual three-dimensional area of ​​the damaged region.

[0159] The spatial distribution and three-dimensional actual area of ​​the damaged area are used as quantitative results of the damage.

[0160] A second embodiment of the present invention also provides a system for quantifying the area of ​​structural damage in a building, comprising:

[0161] The acquisition unit is configured to acquire initial panoramic images and corresponding lidar point cloud data of the building structure before it is damaged, as well as multiple initial panoramic images and corresponding lidar point cloud data of the target area after the damage.

[0162] The correction unit is configured to use fisheye distortion correction constrained by the geometric consistency of the structural straight lines to correct the initial panoramic images before and after the damage, thereby obtaining optimized panoramic images before and after the damage.

[0163] The registration unit is configured to select multiple optimized panoramic images after destruction and an optimized panoramic image before destruction corresponding to the target area, and perform registration to obtain an optimized registered panoramic image.

[0164] The fusion unit is configured to project the lidar point cloud data onto the plane of the optimized registration panoramic image based on the optimized registration panoramic image and the corresponding lidar point cloud data to obtain the fusion result;

[0165] The output unit is configured to obtain the segmentation result of the damaged target region based on a preset multi-scale target detection model, and to obtain the quantitative result of the damage to the damaged target region by combining the segmentation result with the fusion result.

[0166] The third embodiment of the present invention also provides an electronic device, the electronic device comprising:

[0167] At least one processor; and,

[0168] The memory is communicatively connected to the at least one processor; wherein,

[0169] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for quantifying the structural damage area of ​​any of the foregoing embodiments.

[0170] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method for quantifying the area of ​​structural damage described in any of the foregoing embodiments.

[0171] The fifth embodiment of the present invention also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the method for quantifying the building structure damage area of ​​any of the foregoing embodiments.

[0172] Figure 17 The illustration shows a method or device 1000 implementing an embodiment of the present invention. In some embodiments, more or fewer devices may be included than illustrated. In some embodiments, it may be implemented using a single or multiple devices. In some embodiments, it may be implemented using cloud-based or distributed devices.

[0173] like Figure 17 As shown, device 1000 includes a processor 1001 for performing various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. Processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0174] The processor and memory described above are used together to execute a program stored in the memory. When the program is executed by a computer, it can implement the methods, steps, or functions described in the above embodiments.

[0175] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, touchscreen, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 17The diagram only shows a portion of the components and does not imply that the device 1000 only includes... Figure 17 The components shown.

[0176] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0177] Although not shown, in this embodiment of the invention, a computer-readable storage medium is provided having a computer program / instructions stored thereon, which, when executed by a processor, implements the method for quantifying the area of ​​structural damage described in the embodiment.

[0178] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0179] Although not shown, embodiments of the present invention also provide a computer program product, including: a computer program / instructions that, when executed by a processor, implement the method for quantifying the area of ​​structural damage described in the embodiments.

[0180] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

Claims

1. A method for quantitatively determining the area of ​​structural damage in a building, characterized in that, Includes the following steps: Step S1: Collect initial panoramic images and corresponding LiDAR point cloud data of the building structure before damage, as well as multiple initial panoramic images and corresponding LiDAR point cloud data of the target area after damage. Step S2: Using fisheye distortion correction constrained by the geometric consistency of the structural straight lines, the initial panoramic images before and after the damage are corrected to obtain optimized panoramic images before and after the damage. Step S3: Select multiple optimized panoramic images after destruction and one optimized panoramic image before destruction corresponding to the target area, and register them to obtain an optimized registered panoramic image; Step S4: Based on the optimized registration panoramic image and the corresponding lidar point cloud data, project the lidar point cloud data onto the plane of the optimized registration panoramic image to obtain the fusion result; Step S5: Based on the preset multi-scale target detection model, obtain the segmentation result of the damaged target region, and combine the segmentation result with the fusion result to obtain the quantitative result of the damage to the damaged target region.

2. The method for quantifying the structural damage area of ​​a building according to claim 1, characterized in that, When performing fisheye distortion correction in step S2, the following steps are included: Step A1: Offline calibration of the panoramic camera is performed using the checkerboard calibration method. Multiple sets of checkerboard images are acquired under various poses and viewpoints, corner features of the checkerboard images are extracted, and the intrinsic parameter matrix of the panoramic camera is solved using nonlinear least squares optimization. With distortion coefficient matrix ; ； ； In the formula, This indicates the focal length of the panoramic camera along the x-axis. This represents the offset of the origin of the image coordinate system along the x-axis relative to the center of the imaging plane; This represents the focal length of the panoramic camera along the y-axis. This represents the offset of the origin of the image coordinate system in the y-axis direction relative to the center of the imaging plane; , and Both represent radial distortion coefficients; and Both represent tangential distortion coefficients; Step A2: Based on the intrinsic parameter matrix and distortion coefficient matrix Perform distortion correction on the initial panoramic image to obtain the initial corrected image. ; Step A3: Extract the initial corrected image using the Canny edge detection operator. The significant edge information is obtained, and the initial corrected image is detected by probabilistic Hough transform. The set of line segments in the structure, along with significant edge information, serves as a reference set for line constraints, forming constraints for the geometric consistency of the structural lines. Step A4: Extract each structural straight line from the linear constraint reference set. pixel set The vertical distance from each pixel to its corresponding fitted line is calculated as the local geometric error. With minimizing the curvature of the line as the optimization objective, the local geometric errors of the line constraint reference set are fitted using the least squares method to construct a global line curvature error function. And based on the global straight line bending error function The distortion coefficient matrix D is optimized; based on the optimized distortion coefficient matrix D and the intrinsic parameter matrix... For the initial corrected image Perform secondary distortion correction; ； In the formula, Represented by the distortion coefficient matrix Controlled anti-distortion mapping function; An ideal linear model fitted by the least squares method; Step A5: When the residual of the straight line error fitting is less than the preset pixel threshold and the residual distortion rate is less than the preset distortion rate threshold, output the optimized panoramic image before and after the destruction.

3. The method for quantifying the structural damage area of ​​a building according to claim 1, characterized in that, When obtaining the optimized registration panoramic image in step S3, the following steps are included: Step B1: Scale-Invariant Feature Transform (SIFT) algorithm is used to extract key points and corresponding feature descriptors from the optimized panoramic images before and after the damage. Nearest neighbor search is then performed using the Euclidean distance of the feature descriptors to establish an initial feature matching set between the optimized panoramic images before and after the damage. : ‌; In the formula, Key points of the optimized panoramic image before damage; Key points of the optimized panoramic image after destruction; Represents the initial feature matching set The total number of matching point pairs in the middle; Represents the initial feature matching set The index of the i-th matching point pair is a positive integer from 1 to N; Step B2: Based on the initial feature matching set Extract the corresponding optimized panoramic image before the damage. And the optimized panoramic image after destruction The significant linear features are identified, and sets of linear lines are constructed for the optimized panoramic images before and after the destruction, including: ； ； In the formula, Indicate quantity; Step B3: Based on the initial feature matching set Each initial matching point pair Based on the consistency of distance and direction to the nearest structural line in the set of lines, initial matching point pairs with higher consistency are selected. Construct an optimized matching set , ; Step B4: Based on optimizing the matching set The keypoint sets of the optimized panoramic image before and after damage are extracted. The homography matrix between the optimized panoramic image before and after damage is estimated using the Random Sample Consensus Algorithm (RANSAC). Based on homography matrix The overall geometric transformation relationship between the optimized panoramic images before and after the destruction is obtained under the assumption of planar projection, and the initial registered panoramic image is obtained based on the overall geometric transformation relationship; Step B5: Calculate the initial matching point pairs The initial matching error and the root mean square error of the initial registered panoramic image are used to determine the actual improvement rate of the root mean square error of the registration. If the actual improvement rate is greater than or equal to the preset improvement rate, the initial registered panoramic image is used as the optimized registered panoramic image; otherwise, the parameters of the Random Sample Consensus Algorithm (RANSAC) are adjusted, and the homography matrix is ​​re-estimated. The system then regenerates the initial registered panoramic image, judges the actual improvement rate, and continues until the actual improvement rate meets the preset requirements or reaches the maximum number of iterations. Finally, it outputs the current optimal registered panoramic image as the optimized registered panoramic image.

4. The method for quantifying the structural damage area of ​​a building according to claim 3, characterized in that, When selecting initial matching points with high consistency, if the initial matching points are consistent... If each initial matching point falls within the neighborhood of a salient linear feature in the optimized panoramic image before and after the destruction, and satisfies the consistency constraint relative to the direction of the corresponding salient linear feature, then the initial matching point pair is determined to be... If a match is valid, it is considered a false match and discarded, based on a valid initial match pair. Construct the optimized matching set .

5. The method for quantifying the structural failure zone of a building according to claim 1, characterized in that, When obtaining the fused depth map in step S4, the following steps are included: Step C1: Perform joint calibration of multiple sensors based on the checkerboard calibration method to obtain the extrinsic parameter relationship between the LiDAR and the panoramic camera; Step C2: Based on the aforementioned extrinsic parameter relationship, perform coordinate transformation on the lidar point cloud data to obtain the point cloud projection coordinates in the panoramic camera coordinate system. : ； In the formula, The homogeneous coordinates of the lidar point cloud data. It is a complete set of lidar point cloud data. a subset of , , and They are respectively Three-dimensional coordinate components in the lidar coordinate system The coordinates are in the horizontal direction. The coordinates are in the vertical direction. The coordinates are in the depth direction. It is a translation vector; It is a rotation matrix; This is the intrinsic parameter matrix; Step C3: Based on the intrinsic parameter matrix K and the panoramic camera fisheye projection model, project the point cloud coordinates. Projecting onto the plane of the optimized registered panoramic image generates visible points. During projection, only the point cloud projection coordinates located in front of the panoramic camera and whose projection coordinates fall within the effective field of view of the optimized registered panoramic image are retained. If multiple point cloud projection coordinates are projected onto the same pixel or adjacent pixel area, the Z-buffer depth caching strategy is used to retain only the point cloud projection coordinates closest to the panoramic camera as visible points. Step C4: Based on the projection position of each visible point in the optimized registration panoramic image, extract the color information of the corresponding pixel in the optimized registration panoramic image, map it to the visible point, generate color point cloud data, and align the visible points with the optimized registration panoramic image at the pixel level. Project the point cloud coordinates of the visible points. The Z-axis coordinates are used as projection depth information. The depth information is organized into dense or semi-dense depth maps according to pixel positions. The color point cloud data and the depth map are the fusion result.

6. The method for quantifying the structural damage area of ​​a building according to claim 1, characterized in that, When obtaining the segmentation result of the damaged target region in step S5, the following steps are included: Step D1: Input the optimized panoramic image into the multi-scale object detection model. The multi-scale object detection model extracts feature representations at different levels through layer-by-layer convolution and downsampling operations to obtain multi-scale feature maps from local texture to high-level semantics. Step D2: Based on the multi-scale context fusion structure, the features under different receptive fields of the multi-scale feature map are encoded and fused in parallel to obtain the candidate detection box and the corresponding confidence of the damaged target region; Step D3: Mark the inner region of the candidate detection box as the potential foreground and the outer region of the candidate detection box as the background to construct the initial mask of the target region after destruction; Step D4: Establish Gaussian mixture models for the pixel color distribution of the initial mask for the foreground and background respectively, and update the segmentation results of the foreground and background under the improved graph cut optimization framework, taking into account the spatial continuity constraints between pixels. When updating the segmentation results of the foreground and background under the improved graph cut optimization framework, the following steps are performed: Step D41: Shrink the initial mask boundary corresponding to the candidate detection box inward by a preset number of pixels to optimize the boundary of the initial mask and generate a binary mask; Step D42: Perform connected component analysis on the binary mask, traverse all foreground connected components, and retain only the connected component with the largest area to generate the foreground mask; Step D43: Perform morphological closing operation on the foreground mask using a 5×5 elliptical structuring element. During the closing operation, use a dilation-then-erosion sequence to fill the tiny holes inside the foreground region until the preset iteration threshold is reached to obtain the segmentation result.

7. The method for quantifying the structural failure zone of a building according to claim 6, characterized in that, When obtaining quantitative results of destruction, the following steps are included: Step E1: Based on the segmentation results, count the number of pixels in the damaged area in the image coordinate system, and the set of pixel coordinates of the damaged area in the image. Use the number of pixels as the pixel area of ​​the damaged area, and the set of pixel coordinates as the pixel spatial distribution position of the damaged area. Step E2: Based on the projection relationship between the LiDAR point cloud and the panoramic camera image in the fusion result, and combined with the pixel area and pixel spatial distribution of the damaged area, the pixel coordinates corresponding to the damaged area are mapped to the three-dimensional space under the panoramic camera coordinate system to obtain the real spatial scale information of the damaged area. Step E3: Extract the point cloud depth data corresponding to the spatial location of pixels in the damaged area, calculate the actual physical scale corresponding to a single pixel, and multiply the pixel scale by the pixel area of ​​the damaged area to obtain the three-dimensional actual area of ​​the damaged area. The spatial distribution and three-dimensional actual area of ​​the damaged area are used as quantitative results of the damage.

8. A system for quantifying the structural failure zone of a building, characterized in that, The method for quantifying the structural damage area of ​​a building according to any one of claims 1-7 includes: The acquisition unit is configured to acquire initial panoramic images and corresponding lidar point cloud data of the building structure before it is damaged, as well as multiple initial panoramic images and corresponding lidar point cloud data of the target area after the damage. The correction unit is configured to use fisheye distortion correction constrained by the geometric consistency of the structural straight lines to correct the initial panoramic images before and after the damage, thereby obtaining optimized panoramic images before and after the damage. The registration unit is configured to select multiple optimized panoramic images after destruction and an optimized panoramic image before destruction corresponding to the target area, and perform registration to obtain an optimized registered panoramic image. The fusion unit is configured to project the lidar point cloud data onto the plane of the optimized registration panoramic image based on the optimized registration panoramic image and the corresponding lidar point cloud data to obtain the fusion result; The output unit is configured to obtain the segmentation result of the damaged target region based on a preset multi-scale target detection model, and to obtain the quantitative result of the damage to the damaged target region by combining the segmentation result with the fusion result.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for quantifying the area of ​​structural damage as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method for quantifying the area of ​​structural damage as described in any one of claims 1 to 7.

Images