Building three-dimensional defect detection method, device and equipment based on multi-sensor fusion

Abstract

The invention belongs to the field of building surveying and mapping, and particularly discloses a building three-dimensional defect detection method, device and equipment based on multi-sensor fusion, and the method comprises the steps: obtaining data detected by a laser radar, an IMU, a wheel speed meter and image capturing equipment; fusing the data observed by the laser radar, the IMU and the wheel speedometer based on the ESKF, and determining the speed and the pose of the vehicle-mounted acquisition equipment; obtaining a three-dimensional point cloud model under a world coordinate system by combining the speed, the pose and the point cloud data observed by the laser radar; generating a corresponding three-dimensional Gaussian function according to the three-dimensional point cloud model, projecting the three-dimensional Gaussian function in a world coordinate system to a coordinate system of an image capturing device in combination with a pose, and performing microrasterization to obtain a preliminarily rendered image; and constructing a loss function in combination with a visual image observed by an image capturing device, optimizing the three-dimensional Gaussian function, and iterating to obtain a rendered three-dimensional visual model. According to the invention, fusion positioning and high-precision mapping of the building are realized in combination with multiple sensors.

Description

Technical Field

[0001] The present application belongs to the field of architectural surveying and mapping, and more specifically, to a method, device and equipment for detecting three-dimensional architectural defects based on multi-sensor fusion. Background Art

[0002] During the construction or renovation process, a series of problems such as concrete cracks, spalling, honeycombed surfaces, structural deformation, and molding size deviation will inevitably occur in building construction projects. While affecting the quality of the building, there are also a large number of safety hazards. Therefore, how to efficiently and accurately detect defects in completed buildings has become an important requirement in the process of house defect detection.

[0003] Currently, the measurement method of building defects mainly relies on manual visual inspection and total station scanning detection, which is inefficient and costly. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a method, device and equipment for three-dimensional building defect detection based on multi-sensor fusion, aiming to solve the problem that the current building defect measurement method mainly relies on manual visual inspection and total station scanning detection, which is low in efficiency and high in cost.

[0005] To achieve the above objectives, in a first aspect, the present application provides a method for detecting three-dimensional building defects based on multi-sensor fusion, comprising:

[0006] Acquire data from at least one laser radar, an inertial measurement unit (IMU), a wheel speed meter, and an imaging device on a vehicle-mounted acquisition device when the building to be inspected is inspected; the building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs inspection inside the building;

[0007] Based on the error state iterative extended Kalman filter (ESKF), the data observed by the laser radar, IMU and wheel speed meter are fused to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the fused data are fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure;

[0008] Combining the speed, posture, and point cloud data observed by the laser radar, a three-dimensional point cloud model of the building to be detected in the world coordinate system is obtained;

[0009] A corresponding three-dimensional Gaussian function is generated according to the three-dimensional point cloud model, and the three-dimensional Gaussian function in the world coordinate system is projected to the coordinate system of the imaging device in combination with the posture, and differentiable rasterization is performed to obtain an image of the three-dimensional point cloud model that is initially rendered; a loss function is constructed in combination with the visual image observed by the imaging device, and the three-dimensional Gaussian function is optimized, and the rendered three-dimensional visual model is obtained iteratively.

[0010] It can be understood that the method provided in the present application can establish an accurate three-dimensional visual model of a building with a hollow long linear structure inside, which can be achieved through automated detection instead of manual labor; the method provided in the present application can quickly establish a three-dimensional point cloud model and a three-dimensional visual model of the building, and rapid detection can be achieved based on the above three-dimensional visual model, which can solve the current problem that building detection relies on manual labor, has low efficiency and high cost.

[0011] In one example, the method further includes:

[0012] Defect detection is performed on the building to be inspected based on its 3D visual model.

[0013] In one example, the back propagation includes:

[0014] The state quantity of the vehicle-mounted acquisition device is solved based on the change of the point cloud data observed by the laser radar, and the observation equation is constructed by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity at least includes: posture;

[0015] Constructing a Jacobian matrix according to the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation;

[0016] Solve the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, take the eigenvector direction corresponding to the eigenvalue as the degenerate direction, decompose the pose of the vehicle-mounted acquisition device calculated based on the current frame point cloud data, and only retain the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

[0017] In one example, a corresponding three-dimensional Gaussian function is generated according to a three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by a spherical harmonic function, and then setting the opacity of the ellipsoid.

[0018] In one example, the differentiable rasterization uses a tile-based rasterizer, the rasterizer includes back propagation to track the accumulated value of the opacity; the opacity is constrained by a sigmoid activation function to ensure gradient smoothing;

[0019] The loss function is constructed by combining L1 loss and structural similarity index terms.

[0020] In one example, constructing the observation equation includes:

[0021] y=h(x t )+V;

[0022] Where y represents the actual measurement value obtained by solving the point cloud data observed by the lidar, and h(x t ) represents the observation equation of the state quantity, x t represents the state quantity after forward propagation prediction, and V represents Gaussian white noise;

[0023] The Jacobian matrix H is:

[0024] The observation matrix is: H T H.

[0025] In one example, the speed and posture of the laser radar and the imaging device are obtained by transforming the speed and posture of the vehicle-mounted acquisition device according to the positional relationship between the laser radar and the imaging device and the vehicle-mounted acquisition device; and / or

[0026] During the backward propagation, the point cloud data of the laser radar that observes the extension direction of the long linear structure is fused with the data after the primary fusion for a second time.

[0027] In a second aspect, the present application provides a building three-dimensional defect detection device based on multi-sensor fusion, comprising:

[0028] The data acquisition unit is used to acquire data when the laser radar, IMU, wheel speed meter and imaging device located on the vehicle-mounted acquisition device detect the building to be detected; the building to be detected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs detection inside it;

[0029] A data fusion unit is used to fuse the data observed by the laser radar, IMU and wheel speed meter based on ESKF to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the once fused data is fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure;

[0030] A point cloud model building unit is used to combine the speed, posture, and point cloud data observed by the laser radar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system;

[0031] The visual model building unit is used to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iterate to obtain a rendered three-dimensional visual model.

[0032] In one example, the data fusion unit, during backward propagation, solves the state quantity of the on-board acquisition device based on the changes in the point cloud data observed by the laser radar, and constructs an observation equation by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity includes at least: posture; constructs a Jacobian matrix according to the partial derivatives of the observation equation relative to the state quantity after forward propagation fusion, and transposes the Jacobian matrix and multiplies it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solves the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the on-board acquisition device calculated based on the point cloud data of the current frame is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

[0033] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any one example of the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any one of the examples of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any one example of the first aspect.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] In general, the above technical solutions conceived by the present application have at least the following beneficial effects compared with the prior art:

[0038] The present application provides a method, device and equipment for detecting three-dimensional defects in buildings based on multi-sensor fusion. It utilizes multi-sensor data such as lidar, inertial measurement unit, wheel speed meter, and imaging device, and proposes a lidar-IMU-wheel speed meter fusion positioning algorithm framework suitable for long-line structure buildings based on ESKF. It also performs degradation processing on the degraded environment of long-line structure buildings, integrates the wheel speed observation information, realizes the functions of high-precision positioning and point cloud map construction in the above-mentioned degradation scenario, and obtains a high-precision three-dimensional point cloud model of the building. In addition, the present application utilizes the above-mentioned positioning information and three-dimensional point cloud model for rendering, utilizes the three-dimensional point cloud model to generate a three-dimensional Gaussian function used for rendering, and utilizes the visual image obtained by the imaging device to construct a loss function, optimizes the position, shape, opacity and color of the three-dimensional Gaussian function, realizes scene rendering through rasterization, realizes the fusion of the three-dimensional point cloud model and the visual image, and obtains the three-dimensional visual model of the building, thereby improving the efficiency and accuracy of three-dimensional reconstruction of the building, and solving the problems of high efficiency and low cost of building defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is an architecture diagram of the vehicle-mounted acquisition device provided in an embodiment of the present application;

[0040] Figure 2 It is a flow chart of a method for detecting three-dimensional building defects based on multi-sensor fusion provided in an embodiment of the present application;

[0041] Figure 3 It is a schematic diagram of the framework of the laser radar-IMU-wheel speed meter fusion positioning algorithm based on the ESKF algorithm provided in the embodiment of the present application;

[0042] Figure 4 is a decomposition schematic diagram of the posture optimization solution provided in the embodiment of the present application;

[0043] Figure 5 is a flow chart of a three-dimensional Gaussian function radiation field rendering algorithm provided in an embodiment of the present application;

[0044] Figure 6 It is a comparison diagram of the motion trajectories of the vehicle-mounted acquisition equipment obtained by the degradation analysis ablation experiment provided in the embodiment of the present application;

[0045] Figure 7 It is a schematic diagram of the comparison of the three-axis positions of the vehicle-mounted acquisition device obtained by the degradation analysis ablation experiment provided in the embodiment of the present application;

[0046] Figure 8 It is a comparison diagram of the rendering effect of the present application scheme provided by the embodiment of the present application and the rendering effect of the prior art; (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP;

[0047] Fig. 9 It is another comparison diagram of the rendering effect of the present application scheme provided in the embodiment of the present application and the rendering effect of the prior art; (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP;

[0048] Fig.10 This is an architecture diagram of a building three-dimensional defect detection device based on multi-sensor fusion provided in an embodiment of the present application;

[0049] Fig.11 It is an architectural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The term "and / or" in this application is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this application indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0052] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0053] First, the technical terms involved in the embodiments of the present application are introduced.

[0054] (1) Long Linear Structure

[0055] The extension direction of its long linear structure generally cannot provide geometric constraints on the structure and belongs to a structurally degenerate environment. Usually, the extension direction of its long linear structure is called the degenerate direction, and the direction perpendicular to it is called the non-degenerate direction.

[0056] (2) Visual Image

[0057] The visual images mentioned in this application are acquired through a depth camera, which includes information such as color and depth.

[0058] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0059] Figure 1 This is a diagram of the architecture of the vehicle-mounted acquisition device provided by the embodiment of the present application; Figure 1 As shown, it includes: a laser radar, a wheel speed meter, an IMU, an imaging device, a vehicle body and wheels; wherein the wheel speed meter is arranged on the wheel and moves synchronously with the wheel; the laser radar, the IMU and the imaging device are all arranged on the vehicle body.

[0060] Optionally, the laser radar may include two, one installed horizontally and the other installed vertically. The above laser radar may be a 16-line laser radar. During the detection process of the vehicle-mounted acquisition equipment, the laser radar must be fully moved and flipped to ensure that each laser radar fully acquires the complete point cloud of the calibration scene.

[0061] Optionally, the imaging device is a depth camera. The camera integrates a stereo image sensor, an infrared signal transmitter, and an RGB sensor, and can capture images and calculate differences between images, collect color data, illuminate objects, and collect depth data.

[0062] Further optionally, the above-mentioned laser radar is used to observe point cloud data of the building to be inspected; the imaging device is used to obtain the visual image of the building; the IMU is used to observe the position and speed of the vehicle-mounted acquisition device; and the wheel speed meter is used to observe the driving speed of the vehicle-mounted acquisition device.

[0063] In an optional scenario, the building to be detected is a long linear structure, taking a tunnel as an example. The above building may have the following characteristics:

[0064] (1) Global Navigation Satellite System (GNSS) denial: GNSS signals cannot be received in tunnels, and positioning and navigation cannot be achieved by GNSS.

[0065] (2) Structural degradation environment. Tunnels are long linear structures with a generally circular cross section and no obvious structural features. They cannot provide geometric constraints along the tunnel's forward direction and are therefore structurally degraded environments.

[0066] (3) Weak lighting. There is no natural light in the tunnel, and even if there are internal electric lights, the interior of the tunnel is still relatively dim.

[0067] (4) Weak texture. Tunnel linings are mostly made of reinforced concrete materials. Usually, there is no texture information in the tunnel lining. Therefore, highway tunnels belong to weak texture environments.

[0068] It is understandable that the solution provided by the present application is applicable to the scene of detecting the interior of a long linear structure building. The characteristics of the above scene are similar to the tunnel scene. To simplify the description, the tunnel scene may be mentioned in the following specific embodiments to specifically illustrate the embodiments of the present application. Those skilled in the art should understand that the tunnel scene should only be used as an example and cannot be used as any limitation to the solution of the present application.

[0069] Figure 2 is a flow chart of a method for detecting three-dimensional building defects based on multi-sensor fusion provided in an embodiment of the present application; Figure 2 As shown, the following steps are included:

[0070] Step S101, obtaining data when at least one laser radar, IMU, wheel speed meter and imaging device located on a vehicle-mounted acquisition device detects a building to be detected; the building to be detected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs detection inside it.

[0071] Step S102, based on ESKF, the data observed by the laser radar, IMU and wheel speed meter are fused to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the once fused data is fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure.

[0072] Those skilled in the art can understand that if one wants to obtain a three-dimensional point cloud model of the building to be measured, one needs to have the point cloud data observed by the lidar and the speed and posture information of the lidar; in the above-mentioned long-linear structure building environment, it is difficult to obtain the above-mentioned speed and posture information through traditional positioning methods, so it is necessary to integrate the lidar, IMU and wheel speed meter data to achieve it.

[0073] In this process, the above-mentioned back propagation includes:

[0074] The state quantity of the vehicle-mounted acquisition device is solved based on the change of the point cloud data observed by the laser radar, and the observation equation is constructed by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity at least includes: posture;

[0075] Constructing a Jacobian matrix according to the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation;

[0076] Solve the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, take the eigenvector direction corresponding to the eigenvalue as the degenerate direction, decompose the pose of the vehicle-mounted acquisition device calculated based on the current frame point cloud data, and only retain the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

[0077] Further, for example, constructing an observation equation includes:

[0078] y=h(x t )+V;

[0079] Where y represents the actual measurement value obtained by solving the point cloud data observed by the lidar, and h(x t ) represents the observation equation of the state quantity, x t represents the state quantity after forward propagation prediction, and V represents Gaussian white noise;

[0080] The Jacobian matrix H constructed based on the partial derivative of the observation equation h with respect to the error state quantity is:

[0081] The above observation matrix is: H T H.

[0082] In a more specific embodiment, based on the ESKF framework, the original data is the speed information obtained by the wheel speed meter, the three-axis acceleration and three-axis angular velocity obtained by the IMU, and the point cloud data obtained by the lidar. By fusing the data of multiple sensors, the 18-dimensional state of the on-board acquisition equipment is estimated, which are six three-dimensional vectors: position, angle, speed, accelerometer zero bias, gyroscope zero bias, and gravity acceleration g.

[0083] The first is the forward propagation process of the fusion of IMU and wheel speed meter. The forward propagation link uses the three-axis acceleration and three-axis angular velocity data of IMU to recurse the kinematic state of the vehicle acquisition equipment state, obtain high-frequency posture information, and then use the wheel speed meter speed information to update the speed value in the state quantity. The backward propagation process uses the posture information in the forward propagation to dedistort the laser point cloud, and then calculates the point cloud residual by iteratively short-cutting the nearest point. The solution that minimizes the point-to-line surface residual is obtained through iterative solution, and the Kalman gain is calculated to update the posture information in the state quantity.

[0084] Specifically, as an example, the forward propagation of IMU and wheel speed meter includes the following two main parts:

[0085] Part 1: Integrating the IMU measurement values ​​can be used for state recursion. Acceleration integration can get velocity, velocity integration can get position, angular velocity integration can get angle, and the corresponding kinematic discrete equations are obtained. Then the kinematic equations are discretized and the error state is propagated:

[0086] δx←f(x,δx,u m , i) = F(x, u m )·δx+F i ·i

[0087] Among them, f(x, δx, u m , i) represents the motion equation of ESKF; vector x represents the predicted state, δx represents the error state vector, u m represents the input vector, including acceleration and angular velocity measurements, i represents the random perturbation vector, including velocity, attitude, acceleration and angular velocity; F(x, u m ) represents f with respect to the error state δ x The Jacobian matrix, F i represents the Jacobian matrix of f with respect to the perturbation vector i.

[0088] The error covariance matrix of the prediction propagation part in the error state Kalman filter is obtained by linearizing the error state kinematics. The error covariance matrix propagation equation is as follows:

[0089]

[0090] Among them, P represents the covariance matrix of the state quantity, F x and F i is f with respect to the error state δ x and the Jacobian matrix of the perturbation vector i, Q i is the covariance matrix of the perturbation vector; ← is the assignment symbol.

[0091] Part 2: When the vehicle-mounted acquisition device is driving normally in the tunnel, the speed of the detection robot along the direction of the vehicle's forward movement can be obtained through the measurement value of the wheel speed meter. Only the speed along the direction of the vehicle's forward movement, when there is no bumping and side sliding, will not produce lateral or vertical speed due to the kinematic constraints of the vehicle-mounted acquisition device. At this time, the three-axis speed value of the vehicle can be obtained.

[0092] In the process of state estimation, the position, velocity and rotation of the state quantity are first obtained by integrating the IMU data. After obtaining the velocity measurement value, the state quantity is updated according to the velocity value. Since the velocity is observed and combined with the kinematic model of the IMU, the inferred posture will not diverge quickly.

[0093] As an example, the above backward propagation includes the following two main parts:

[0094] Part I: The observation model can be written as follows:

[0095] y=h(x t )+V

[0096] Among them, h(x t ) is the observation equation, and V is Gaussian white noise.

[0097] Calculate the Kalman gain according to the observation equation, update the state, and propagate the error to obtain:

[0098] K=PH T (HPH T +V) -1

[0099]

[0100] P←(I-KH)P

[0101] Among them, K is the Kalman gain, P is the predicted state quantity, Represents the optimal estimated update value of the state quantity; Jacobian matrix

[0102] Update the forecast status as follows:

[0103]

[0104] Where P, v, q, a b ,ω b They respectively represent the position, velocity, rotation, accelerometer bias, and gyroscope bias after the status is updated. represents the optimal estimate of each component in the error state, and θ represents the rotation angle.

[0105] Specifically, the actual measurement value is obtained by solving the point cloud data observed by the lidar based on the iterative closest point algorithm (ICP). The key to the ICP algorithm is to solve the precise pose by continuously iterating to find corresponding points in the source point cloud and the target point cloud and minimizing the residual between them.

[0106] Assume that the current number of iterations of the iterative Kalman filter is k, and transform the point cloud dedistorted by backpropagation into the world coordinate system:

[0107]

[0108] Among them, the superscript G represents the world coordinate system; It indicates the expected coordinate value of the feature point after the pose to be solved is transformed into the world coordinate system; I T L Represents the transformation from the radar laser coordinate system to the IMU coordinate system; Represents the coordinate value of the feature point in the kth frame radar laser coordinate system; represents the pose to be solved; f j represents the jth feature point; I k represents the kth frame IMU; m represents the total number of feature points.

[0109] Specifically, G is the distance from the feature point to the matching feature (line or plane) to generate the residual. For edge feature points, the residual is the distance from the point to the line; for plane feature points, the residual is the distance from the point to the plane. Define G i is the normal vector of the plane or edge, G q j is a point on a plane or an edge, then the residual It is expressed as:

[0110]

[0111] Among them, the above residual It is the residual function of the lidar point cloud observation.

[0112] By solving the observation equation linearly, the observation equation of the extended Kalman filter is derived. Subsequently, multiple iterative optimizations are required to reduce the error caused by the linearized observation equation. It can be proved that the solution method of the iterative extended Kalman filter and the optimization method based on Newton's method are mathematically equivalent, and can solve the problem with an unbiased estimate.

[0113] During the iteration process, if the residual is less than the threshold, the iteration is considered to have converged, and the optimal state quantity is estimated as:

[0114]

[0115] The state quantity X k Update and convert to Lie group form Each lidar point cloud in the kth scanning frame The map is updated by projecting everything into the world coordinate system using the following formula.

[0116]

[0117] Through iteration, the tightly coupled state estimation of LiDAR and IMU is achieved. The posture of the tunnel detection machine and the zero bias of the IMU can be updated through the observation of LiDAR, achieving more accurate state estimation and global point cloud.

[0118] Part II: In the simultaneous localization and mapping problem, degradation refers to the phenomenon that the algorithm may perform poorly in an environment with insufficient constraints due to the instability and inaccuracy of state estimation, such as insufficient observation information in a certain direction or dimension, making it difficult for the algorithm to accurately estimate the state from the observation data.

[0119] LiDAR requires spatial geometric structure information to iteratively solve the pose information. Since the long linear structure lacks structural features in its extension direction and belongs to a degenerate environment, the state estimation is difficult to estimate along the long linear extension method due to the lack of constraints. It must be degenerated and the solved pose is projected to the non-degenerate method. The components of the solution in the degenerate direction are discarded and the components in the non-degenerate direction are retained.

[0120] There is an observation matrix in the solution process. T The degradation factor D can be calculated by eigendecomposing H as follows:

[0121] D=λ min +1

[0122] Perform feature decomposition on the point cloud feature matrix, λ min H T The minimum eigenvalue of H. If the minimum eigenvalue is less than the threshold, the direction of the eigenvector corresponding to the eigenvalue corresponds to the direction of degradation.

[0123] Define the following matrix:

[0124]

[0125] Among them, V p represents the degradation direction vector; V u represents the non-degenerate direction vector; V f represents all direction vectors obtained by the solution; n represents the total number of solution directions.

[0126] The corresponding reprojection components are:

[0127]

[0128] in, is an estimate of the true state, is the estimated value obtained after nonlinear optimization, For the optimal solution, x′ p represents the increment of the solution in the degenerate direction; x′ u represents the increment of the solution in the non-degenerate direction.

[0129] like Figure 4 As shown, the solution is decomposed into degenerate and non-degenerate directions, represented by orange and blue, respectively.

[0130] Keeping the solution in the non-degenerate direction, the final solution is:

[0131]

[0132] Where Δx u Represents the optimal estimated value of the state quantity after updating.

[0133] Further optionally, during backward propagation, the point cloud data of the laser radar observing the extension direction of the long linear structure (that is, only the point cloud data of the laser radar in this direction is needed for secondary fusion to update the posture information) is fused for the second time with the data after the first fusion.

[0134] Step S103, combining the speed, posture, and point cloud data observed by the laser radar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system.

[0135] It can be understood that the speed and posture used in step S103 are the speed and posture of the laser radar, which can be obtained by transforming the speed and posture of the vehicle-mounted collection device according to the positional relationship of the laser radar relative to the vehicle-mounted collection device.

[0136] It should be noted that in order to make the three-dimensional point cloud model more accurate, this step combines the point cloud data observed by all lidars to construct the three-dimensional point cloud model.

[0137] In this application, the vehicle-mounted acquisition device chooses to use laser radar as the main sensor to realize the positioning and mapping functions of the robot. However, there are many limitations in using only laser radar for positioning and mapping, because the lack of the ability to detect the inertial state of the vehicle-mounted acquisition device itself leads to large deviations in the point cloud measurement results when the vehicle-mounted acquisition device moves at a fast speed or there are bumps. IMU can measure the acceleration and angular velocity of the vehicle-mounted acquisition device, and can recurse the prior state based on the IMU data, dedistort the laser radar point cloud and use it as the initial value of the point cloud matching, which can improve the accuracy and robustness of positioning in mapping. The wheel speed meter can provide speed observations for the vehicle-mounted acquisition device, which is used to update the speed state quantity in the state estimation. When the tunnel point cloud is degraded, the wheel speed meter provides a state estimate in the degradation direction, making the positioning more robust. Therefore, this application will integrate IMU, laser radar and wheel speed meter data to realize the positioning and mapping functions of the tunnel environment.

[0138] Step S104, generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iterate to obtain the rendered three-dimensional visual model.

[0139] It can be understood that the posture used in step S104 is the posture of the imaging device, which can be obtained by transforming the speed and posture of the vehicle-mounted acquisition device according to the positional relationship of the imaging device relative to the vehicle-mounted acquisition device.

[0140] In one example, a corresponding three-dimensional Gaussian function is generated according to a three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by a spherical harmonic function, and then setting the opacity of the ellipsoid.

[0141] In one example, the differentiable rasterization uses a tile-based rasterizer, the rasterizer includes back propagation to track the accumulated value of the opacity; the opacity is constrained by a sigmoid activation function to ensure gradient smoothing;

[0142] The loss function is constructed by combining L1 loss and structural similarity index terms.

[0143] In a specific embodiment, the three-dimensional point cloud model constructed in step S103 and the depth image obtained in step S101 are used as input, and a 3D Gaussian function is used as a scene representation. Compared with traditional point clouds, the 3D Gaussian function provides more continuity and smoothness. The quality of the scene representation is improved by optimizing the properties of the 3D Gaussian function, iteratively optimizing the properties such as the position, opacity, covariance and spherical harmonic coefficients of the 3D Gaussian function, and an adaptive density control step is used to dynamically increase or decrease the number of 3D Gaussian functions. Through adaptive density control, the method is able to generate compact and accurate scene representations. Finally, the present application uses a fast differentiable rasterization method, which utilizes the differentiability of the 3D Gaussian function and the ability to project onto a two-dimensional plane to efficiently mix anisotropic projections and opacity, thereby ensuring the correct visibility order. In addition, by constructing a residual between the rendered image obtained by rasterization and the original image, fast and accurate back propagation is achieved. For a specific algorithm flowchart, see Figure 5 shown.

[0144] The core of radiation field rendering based on 3D Gaussian function is 3D Gaussian function parameter optimization and adaptive density control of 3D Gaussian function. Through iterative loops, the algorithm continuously samples training views to obtain camera posture and image information. Then, use this information to project the points into the image to generate a rasterized image. Next, the loss function between the generated image and the reference image is calculated to measure the difference between the images. The loss function is back-propagated and parameter updated to adjust the position, covariance, color, and transparency attributes of the point cloud to improve image quality.

[0145] If the refinement iteration conditions are met, the following operations are performed:

[0146] (1) For each Gaussian distribution (represented by Gaussian mean, covariance, weight and transparency), check whether the transparency is lower than the threshold or whether the mean and covariance are too large. If the conditions are met, prune and remove the Gaussian distribution.

[0147] (2) If the loss gradient with respect to the position exceeds the threshold, a densification operation is performed. If the covariance norm of the point exceeds the threshold, a super-reconstruction operation is performed, that is, the Gaussian distribution is split. Otherwise, an under-reconstruction operation is performed, that is, the Gaussian distribution is cloned.

[0148] (3) Finally, the iteration counter is incremented and the next cycle continues until the algorithm converges. Through iterative optimization and dynamic adjustment of Gaussian distribution properties, the algorithm can improve the accuracy and fidelity of image generation, thus playing an important role in scene rendering.

[0149] In order to enable high-quality iterative optimization of scene representation and achieve high-quality new perspective synthesis based on sparse point cloud models, the present application uses 3D Gaussian functions as the basic spatial representation unit. This method has differentiable volume representation properties and is an unstructured explicit representation that can be rendered quickly. The point cloud is initialized as a set of 3D Gaussian functions to characterize the point cloud as an ellipsoid. The position of the ellipsoid is represented by the mean of the 3D Gaussian function, the shape is represented by the covariance of the 3D Gaussian function, and the color is represented by the spherical harmonic function. The opacity of the ellipsoid is then set. These parameters can be used to create and optimize the radiation field through iterative optimization steps. The key to the efficiency of the method of the present application is that the present application uses a tile-based rasterizer, which allows alpha blending of anisotropic spots and maintains visibility order through fast sorting. In addition, the fast rasterizer of the present application also includes fast backpropagation, by tracking the accumulated alpha values ​​without limiting the number of Gaussian functions that can receive gradients.

[0150] The present application models the geometry as a set of 3D Gaussian functions without normals, which are defined by the center point (mean) μ of the complete 3D covariance matrix Σ defined in world space. During the rasterization projection process of the present application, this Gaussian function is multiplied by the opacity α.

[0151] To avoid the computational overhead caused by automatic differentiation during training, the present application explicitly derives gradients for all parameters. The above-mentioned easy-to-optimize anisotropic covariance representation enables the present application to optimize the 3D Gaussian function to adapt to capture the geometric relationship of different shapes in the scene, thereby obtaining an accurate scene representation.

[0152] Furthermore, in the parameter optimization process, the parameters of the 3D Gaussian function are optimized, including the position p, opacity α, and covariance Σ. In addition, the coefficients of the spherical harmonic function representing the color of each Gaussian ellipsoid are optimized to ensure that the color appearance of the scene is correctly captured. In the process of parameter optimization, the number of 3D Gaussian functions is adaptively controlled to better characterize the scene.

[0153] During the optimization process, the rendering result is compared with the input training view in the data, and the residual is constructed and back-propagated. However, due to the ambiguity of 3D to 2D projection, the ellipsoid may be generated incorrectly, so the optimization algorithm of this application needs to have the ability to generate, delete or move the ellipsoid to correct the position error. In compact scene representation, the quality of the covariance parameters of the 3D Gaussian distribution is critical, because a small number of large anisotropic Gaussian distributions have better performance in large uniform areas. Stochastic gradient descent technology is used for optimization, making full use of the standard GPU acceleration framework, and custom CUDA kernels can be added to achieve fast rasterization of this application. This step is critical to the efficiency of the optimization because it is the main computational bottleneck in the optimization process.

[0154] This application chooses to use the sigmoid activation function to constrain the parameter α to the range [0-1) and ensure gradient smoothness, while using the exponential activation function to control the covariance scale. This application provides a suitable starting state for the model by estimating the initial covariance matrix as an isotropic Gaussian matrix with an axis length equal to the average distance to the three nearest points. The standard exponential decay scheduling technique is used to adjust the position parameters. The loss function designed by the method of this application combines the L1 loss and the structural similarity index (D-SSIM) term, balancing the two by weighing the parameter λ:

[0155]

[0156] First, an initial sparse point set is obtained from simultaneous localization and mapping, and then optimized by adaptively controlling the number and density of Gaussian points per unit volume. This allows the application to transition from the initial sparse Gaussian set to a denser set that better represents the scene and has the correct parameters. After the initial optimization, the application performs densification every 100 iterations and removes any Gaussian distribution that is essentially transparent, that is, its transmittance α is less than the set threshold.

[0157] The goal of adaptive Gaussian control is to fill in blank areas, especially areas lacking geometric features, i.e. under-reconstructed areas. At the same time, this method is also applicable to areas that are covered by a large area of ​​Gaussian distribution, i.e. over-reconstructed areas. There are large view space gradients in both under-reconstructed and over-reconstructed areas, essentially because the reconstruction of the area is incomplete. In the optimization process, the reconstruction of the area is improved by adaptively controlling the number of Gaussian functions.

[0158] This application uses the amplitude of the average view space position gradient exceeding the threshold to perform Gaussian function densification. In the experiment, this application sets the threshold to 0.0002. For the small Gaussian function located in the under-reconstructed area, the existing Gaussian function is copied to create a new Gaussian function, and it is moved along the direction of the position gradient. In the large Gaussian function in the over-reconstructed area, it is decomposed into smaller Gaussian functions, and the original Gaussian function is replaced by two new Gaussian functions, and its scale is reduced by φ = 1.6 times. The reduction factor is the optimal parameter selected by multiple experiments.

[0159] Through the above, the density and distribution of Gaussian functions can be more effectively controlled, thereby improving the quality and accuracy of scene reconstruction. In the under-reconstructed area, the present application detects and handles the need to increase the total volume of the system and the number of Gaussians. In the over-reconstructed area, the present application keeps the total volume unchanged but increases the number of Gaussians. The optimization process of the present application may be affected by the wrongly constructed suspended Gaussian function close to the input camera, which may cause the Gaussian density to be unable to increase reasonably. In order to effectively control the increase in the number of Gaussians, the present application adopts a strategy to set the α value to a value close to zero after every 3000 iterations. Then, those Gaussian functions with greater opacity are added as needed, while those Gaussian functions with transmittance below the threshold are deleted. These Gaussian functions may shrink or expand, and there may be considerable overlapping areas with other Gaussian functions, but by regularly removing those Gaussian functions that are very large in world space or occupy a large area in view space, the overall situation of the number of Gaussians can be well controlled. The Gaussian functions in the model of the present application are always kept in Euclidean space as primitives, and unlike other conventional rendering methods, this method does not require spatial compression or projection of distant or large Gaussian functions.

[0160] In order to achieve fast overall rendering and sorting to support efficient transparency blending, this application designs a fast rasterization method based on a tile-based Gaussian ellipsoid, which is inspired by recent software rasterization methods and pre-sorts the entire image, avoiding the huge amount of calculation caused by sorting each pixel. The fast rasterizer of this application can efficiently perform backpropagation of any number of mixed Gaussian functions, and only requires a constant memory for each pixel, and the additional memory consumption is low. The rasterization process of this application is fully differentiable, and based on the 2D projection, anisotropic 3D Gaussian functions can be rendered.

[0161] The method of the present application first divides the image into 16×16 tiles, and then eliminates the 3D Gaussian function according to the view cone and each tile. The method only retains the Gaussian function with a confidence interval of 99% that intersects the view cone. According to the amount of overlap between the Gaussian function and the tile, a key is assigned to each Gaussian function during the instantiation process, which combines the visual space depth and the tile encoding. The Gaussian functions are sorted using these keys using a fast GPU Radix sort. In the absence of additional pixel-level sorting, transparency blending is based on the initial sorting. Therefore, in some configurations, the transparency blending of the present application may be approximate, but as the spots gradually approach the size of a single pixel, this approximation becomes negligible. The present application found that this method significantly improves training and rendering performance, and does not produce visible artifacts in converged scenes.

[0162] After sorting the Gaussian functions, the application generates a list for each tile, ensuring that the first and last entries in the depth-sorted entries intersect with the given tile. During rasterization, the application launches a thread block for each tile, each of which first loads the Gaussian function data packet into shared memory, and then for a given pixel, traverses the list from front to back to accumulate color and opacity, thereby maximizing the parallelism of data loading, sharing, and processing. When the pixel reaches the target saturation, the corresponding thread stops. The threads in the tile are queried periodically, and when all pixels reach saturation, that is, the opacity approaches 1, the processing of the entire tile is terminated.

[0163] Furthermore, the above Figure 1 The method flow may further include step S105, performing defect detection on the building to be inspected based on the three-dimensional visual model of the building to be inspected.

[0164] Among them, the specific algorithm of the above-mentioned defect detection can select an existing mature defect detection algorithm, and can be more finely selected according to the type of building, which will not be elaborated here.

[0165] Specifically, for the degradation analysis mentioned in step S102, the present application performs degradation analysis based on the structural degradation environment of the tunnel, performs ablation experiments on the degradation analysis part, and draws trajectory comparisons and three-axis position curves including and excluding degradation analysis, respectively. Figure 6 and Figure 7 .from Figure 6 It can be seen from the figure that when the degradation analysis is not included, the algorithm trajectory is not closed, which means that when the degradation analysis is not performed, the algorithm cannot correctly estimate the state. Figure 7 As shown, when no degradation analysis is performed, a good estimation can be made in the direction perpendicular to the tunnel (X direction), and the error between the method of the present application is small, but the error in the direction along the tunnel is large. This is because when no degradation analysis is performed, due to the degradation of the tunnel point cloud data, only the lateral movement can be constrained, resulting in the unobservable longitudinal movement. The wrong longitudinal motion observation leads to the wrong position estimation in the forward direction. Therefore, it can be concluded that the degradation processing performed by the present application plays a key role in the algorithm's ability to achieve accurate positioning in the degraded environment of long linear structure buildings.

[0166] Figure 8 and Fig. 9 These are two comparison diagrams of the rendering effect of the present application scheme provided by the embodiment of the present application and the rendering effect of the prior art; in the two diagrams, (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP; Figure 8As shown in the figure, in the rendering results of the interior of the building, the algorithm of this application can better adapt to the light and dark changes inside the tunnel. The rectangular edge of the tunnel exit at the far end of the tunnel is not clear enough, but overall it has achieved good results. However, the rendering effect of InstantNGP inside the tunnel is poor, and a large number of black shadows are generated inside the tunnel, and a large number of details are missing. Fig. 9 As shown, in the rendering results of the internal facilities of the building, both the algorithm of the present application and InstantNGP achieved good rendering effects and were able to clearly render the internal facilities, but the method of the present application achieved clearer results in the facilities in the red frame area, and the rendering results of InstantNGP were slightly blurred.

[0167] In summary, the accuracy and clarity of the three-dimensional visual model rendered using the solution of the present application are greatly improved compared to the prior art.

[0168] Fig.10 : is a diagram of the architecture of a building three-dimensional defect detection device based on multi-sensor fusion provided in an embodiment of the present application; Fig.10 As shown, including:

[0169] The data acquisition unit 1010 is used to acquire data when the laser radar, IMU, wheel speed meter and imaging device located on the vehicle-mounted acquisition device detect the building to be detected; the building to be detected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs detection inside it;

[0170] The data fusion unit 1020 is used to fuse the data observed by the laser radar, IMU and wheel speed meter based on ESKF to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the once fused data is fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure;

[0171] The point cloud model building unit 1030 is used to combine the speed, posture, and point cloud data observed by the laser radar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system;

[0172] The visual model building unit 1040 is used to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iterate to obtain a rendered three-dimensional visual model.

[0173] Further optionally, the data fusion unit 1020, during backward propagation, solves the state quantity of the on-board acquisition device based on the change of the point cloud data observed by the laser radar, and constructs an observation equation by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity includes at least: posture; constructs a Jacobian matrix according to the partial derivatives of the observation equation relative to the state quantity after forward propagation fusion, and transposes the Jacobian matrix and multiplies it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solves the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the on-board acquisition device calculated based on the point cloud data of the current frame is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

[0174] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program unit in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0175] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, such as Fig.11 As shown, the electronic device includes a processor 1111, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1111, the communication interface 1120 and the memory 1130 communicate with each other via the communication bus 1140. The processor 1111 can call the logic instructions in the memory 1130 to execute the method in the above embodiment.

[0176] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0177] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0178] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0179] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0180] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0182] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0183] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for detecting three-dimensional building defects based on multi-sensor fusion, characterized in that: include: Acquire data from at least one laser radar, an inertial measurement unit (IMU), a wheel speed meter, and an imaging device on a vehicle-mounted acquisition device when inspecting a building to be inspected; The building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted collection equipment performs inspection inside it; Based on the error state iterative extended Kalman filter ESKF, the data observed by the laser radar, IMU and wheel speed meter are fused to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the fused data is fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure; Combining the speed, posture, and point cloud data observed by the laser radar, a three-dimensional point cloud model of the building to be detected in the world coordinate system is obtained; A corresponding three-dimensional Gaussian function is generated according to the three-dimensional point cloud model, and the three-dimensional Gaussian function in the world coordinate system is projected to the coordinate system of the imaging device in combination with the posture, and differentiable rasterization is performed to obtain an image of the three-dimensional point cloud model that is initially rendered; a loss function is constructed in combination with the visual image observed by the imaging device, and the three-dimensional Gaussian function is optimized, and the rendered three-dimensional visual model is obtained iteratively.

2. The method according to claim 1, characterized in that Also includes: Defect detection is performed on the building to be inspected based on its 3D visual model.

3. The method according to claim 1, characterized in that The back propagation includes: The state quantity of the vehicle-mounted acquisition device is solved based on the change of the point cloud data observed by the laser radar, and the observation equation is constructed by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity at least includes: posture; Constructing a Jacobian matrix according to the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation; Solve the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, take the eigenvector direction corresponding to the eigenvalue as the degenerate direction, decompose the pose of the vehicle-mounted acquisition device calculated based on the current frame point cloud data, and only retain the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

4. The method according to claim 1, characterized in that: Generate a corresponding three-dimensional Gaussian function according to the three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by the spherical harmonic function, and then setting the opacity for the ellipsoid.

5. The method according to claim 4, characterized in that The differentiable rasterization uses a tile-based rasterizer, which includes back-propagation to track the accumulated opacity value; the opacity is constrained by a sigmoid activation function to ensure gradient smoothing; The loss function is constructed by combining L1 loss and structural similarity index terms.

6. The method according to claim 3, characterized in that The construction of the observation equation comprises: y=h(x t )+V; Where y represents the actual measurement value obtained by solving the point cloud data observed by the lidar, and h(x t ) represents the observation equation of the state quantity, x t represents the state quantity after forward propagation prediction, and V represents Gaussian white noise; The Jacobian matrix H is: The observation matrix is: H T H.

7. The method according to any one of claims 1 to 6, characterized in that: The speed and posture of the laser radar and the imaging device are obtained by transforming the speed and posture of the vehicle-mounted acquisition device according to the positional relationship between the laser radar and the imaging device and the vehicle-mounted acquisition device; and / or During the backward propagation, the point cloud data of the laser radar that observes the extension direction of the long linear structure is fused with the data after the primary fusion for a second time.

8. A three-dimensional building defect detection device based on multi-sensor fusion, characterized in that: include: A data acquisition unit is used to acquire data from the laser radar, IMU, wheel speed meter and imaging device on the vehicle-mounted acquisition device when inspecting the building to be inspected; The building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted collection equipment performs inspection inside it; A data fusion unit is used to fuse the data observed by the laser radar, IMU and wheel speed meter based on ESKF to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the data of the IMU and the wheel speed meter are fused once through forward propagation, and then the once fused data is fused twice with the point cloud data observed by the laser radar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the laser radar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the laser radar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure; A point cloud model building unit is used to combine the speed, posture, and point cloud data observed by the laser radar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system; The visual model building unit is used to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iterate to obtain a rendered three-dimensional visual model.

9. The device according to claim 8, characterized in that The data fusion unit, during backward propagation, solves the state quantity of the vehicle-mounted acquisition device based on the change of the point cloud data observed by the laser radar, and constructs an observation equation by combining the solved state quantity with the state quantity after forward propagation fusion; The state quantity includes at least: posture; constructing a Jacobian matrix based on the partial derivatives of the state quantity after the observation equation is fused with respect to the forward propagation, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solving the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

10. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

Images