Method, device and equipment for detecting three-dimensional building defects based on multi-sensor fusion

Abstract

This application belongs to the field of building surveying and mapping, and specifically discloses a method, device, and equipment for detecting three-dimensional building defects based on multi-sensor fusion. The method includes: obtaining data detected by a laser radar, an IMU, a wheel speed meter, and an imaging device; fusing the data observed by the laser radar, IMU, and wheel speed meter based on ESKF to determine the speed and posture of the vehicle-mounted acquisition device; combining the speed, posture, and point cloud data observed by the laser radar to obtain a three-dimensional point cloud model in the world coordinate system; generating a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, projecting the three-dimensional Gaussian function in the world coordinate system into the coordinate system of the imaging device in combination with the posture, and performing differentiable rasterization to obtain a preliminary rendered image; constructing a loss function based on the visual image observed by the imaging device, optimizing the three-dimensional Gaussian function, and iteratively obtaining a rendered three-dimensional visual model. Through this application, fusion positioning and high-precision mapping of buildings are achieved by combining multiple sensors.

Description

Technical Field

[0001] The present application belongs to the field of architectural surveying and mapping, and more specifically, relates to a method, device and equipment for detecting three-dimensional architectural defects based on multi-sensor fusion. Background Art

[0002] During the construction or renovation process, building construction projects will inevitably encounter a series of problems such as concrete cracks, spalling, honeycombing, structural deformation, and molding size deviation. These problems not only affect the quality of the building but also pose a large number of safety hazards. Therefore, how to efficiently and accurately detect defects in completed buildings has become an important requirement in the process of house defect detection.

[0003] Currently, the measurement method of building defects mainly relies on manual visual inspection and total station scanning detection, which is inefficient and costly. Summary of the Invention

[0004] In response to the defects of the existing technology, the purpose of this application is to provide a method, device and equipment for three-dimensional building defect detection based on multi-sensor fusion, aiming to solve the problem that the current building defect measurement method mainly relies on manual visual inspection and total station scanning detection, which is low in efficiency and high in cost.

[0005] To achieve the above objectives, in a first aspect, the present application provides a method for detecting three-dimensional building defects based on multi-sensor fusion, comprising:

[0006] Acquire data from at least one laser radar, inertial measurement unit (IMU), wheel speed meter, and imaging device on a vehicle-mounted data acquisition device during inspection of a building to be inspected; the building to be inspected is a long, hollow structure within which the vehicle-mounted data acquisition device performs inspections;

[0007] The data from the LiDAR, IMU, and wheel speedometer observations are fused based on the Error State Iterative Extended Kalman Filter (ESKF) to determine the speed and posture of the on-board acquisition device. The IMU and wheel speedometer data are fused once through forward propagation, and then the fused data is fused twice with the point cloud data observed by the LiDAR through backward propagation. During the backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the LiDAR and the historical map is less than a preset threshold, the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion. The historical map is a point cloud set in the world coordinate system determined based on the historical frame point cloud data observed by the LiDAR, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure.

[0008] Combining the speed, posture, and point cloud data observed by the lidar, a three-dimensional point cloud model of the building to be inspected in a world coordinate system is obtained;

[0009] A corresponding three-dimensional Gaussian function is generated according to the three-dimensional point cloud model, and the three-dimensional Gaussian function in the world coordinate system is projected to the coordinate system of the imaging device in combination with the posture, and differentiable rasterization is performed to obtain a preliminary rendered image of the three-dimensional point cloud model; a loss function is constructed in combination with the visual image observed by the imaging device, and the three-dimensional Gaussian function is optimized, and the rendered three-dimensional visual model is obtained iteratively.

[0010] It can be understood that the method provided in this application can establish an accurate three-dimensional visual model of a building with a hollow long linear structure inside, which can be achieved through automated inspection instead of manual labor; the method provided in this application can quickly establish a three-dimensional point cloud model and a three-dimensional visual model of the building, and rapid inspection can be achieved based on the above three-dimensional visual model, which can solve the current problem that building inspection relies on manual labor, is inefficient and costly.

[0011] In one example, the method further includes:

[0012] Defect detection is performed on the building to be inspected based on its 3D visual model.

[0013] In one example, the back propagation includes:

[0014] Calculate the state of the vehicle-mounted acquisition device based on the changes in the point cloud data observed by the lidar, and construct an observation equation by combining the calculated state and the state after forward propagation fusion; the state includes at least: posture;

[0015] Constructing a Jacobian matrix based on the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation;

[0016] The minimum eigenvalue of the observation matrix is ​​solved. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the pose of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, retaining only the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

[0017] In one example, a corresponding three-dimensional Gaussian function is generated according to a three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by the spherical harmonic function, and then the opacity is set for the ellipsoid.

[0018] In one example, the differentiable rasterization uses a tile-based rasterizer, the rasterizer including backpropagation to track the accumulated opacity value; constraining the opacity by a sigmoid activation function to ensure gradient smoothing;

[0019] The loss function is constructed by combining L1 loss and structural similarity index terms.

[0020] In one example, constructing the observation equation includes:

[0021] y=h(x t )+V;

[0022] Among them, y represents the actual measurement value obtained by solving the point cloud data based on lidar observation, h(x t ) represents the observation equation of the state quantity, x t Represents the state quantity after forward propagation prediction, and V represents Gaussian white noise;

[0023] The Jacobian matrix H is:

[0024] The observation matrix is: H T H.

[0025] In one example, the speed and posture of the laser radar and the imaging device are obtained by transforming the speed and posture of the vehicle-mounted collection device according to the positional relationship between the laser radar and the imaging device relative to the vehicle-mounted collection device; and / or

[0026] During the backward propagation, the point cloud data of the laser radar that observes the extension direction of the long linear structure is fused with the data after the primary fusion for a second time.

[0027] In a second aspect, the present application provides a three-dimensional building defect detection device based on multi-sensor fusion, comprising:

[0028] A data acquisition unit is used to acquire data from the laser radar, IMU, wheel speed meter, and imaging device located on the vehicle-mounted acquisition device when inspecting the building to be inspected; the building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs inspections inside it;

[0029] A data fusion unit is used to fuse the data observed by the lidar, IMU, and wheel speed meter based on the ESKF to determine the speed and posture of the on-board acquisition device; wherein, the data of the IMU and wheel speed meter are fused once through forward propagation, and then the fused data is fused twice with the point cloud data observed by the lidar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the lidar and the historical map is less than a preset threshold, the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined based on the historical frame point cloud data observed by the lidar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure;

[0030] A point cloud model building unit is used to combine the speed, posture, and point cloud data observed by the lidar to obtain a three-dimensional point cloud model of the building to be inspected in the world coordinate system;

[0031] The visual model establishment unit is used to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain an image of the three-dimensional point cloud model that is initially rendered; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iteratively obtain the rendered three-dimensional visual model.

[0032] In one example, the data fusion unit solves the state quantity of the on-board acquisition device based on the changes in the point cloud data observed by the lidar during backward propagation, and constructs an observation equation by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity includes at least: posture; constructs a Jacobian matrix based on the partial derivatives of the observation equation relative to the state quantity after forward propagation fusion, and transposes the Jacobian matrix and multiplies it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solves the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

[0033] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the method described in the first aspect or any one of the examples of the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any one of the examples of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any one of the examples of the first aspect.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] In general, the above technical solutions conceived by this application have at least the following beneficial effects compared with the prior art:

[0038] The present application provides a method, device and equipment for three-dimensional building defect detection based on multi-sensor fusion. It utilizes multi-sensor data such as lidar, inertial measurement unit, wheel speed meter, imaging device, etc., and proposes a lidar-IMU-wheel speed meter fusion positioning algorithm framework suitable for long-line structure buildings based on ESKF. It also performs degradation processing on the degraded environment of long-line structure buildings, integrates the wheel speed observation information, and realizes the functions of high-precision positioning and point cloud map construction in the above-mentioned degraded scenario, thereby obtaining a high-precision three-dimensional point cloud model of the building; in addition, the present application utilizes the above-mentioned positioning information and three-dimensional point cloud model for rendering, uses the three-dimensional point cloud model to generate a three-dimensional Gaussian function used for rendering, and uses the visual image obtained by the imaging device to construct a loss function, optimizes the position, shape, opacity and color of the three-dimensional Gaussian function, realizes scene rendering through rasterization, realizes the fusion of the three-dimensional point cloud model and the visual image, and obtains a three-dimensional visual model of the building, thereby improving the efficiency and accuracy of three-dimensional reconstruction of the building, and solving the problems of high efficiency and low cost of building defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is an architectural diagram of the vehicle-mounted data collection device provided in an embodiment of the present application;

[0040] Figure 2 This is a flow chart of a method for three-dimensional building defect detection based on multi-sensor fusion provided in an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the framework of the LiDAR-IMU-wheel speedometer fusion positioning algorithm based on the ESKF algorithm provided in an embodiment of the present application;

[0042] Figure 4 Schematic diagram of the decomposition of the posture optimization solution provided by the embodiment of the present application;

[0043] Figure 5 This is a flow chart of a three-dimensional Gaussian function radiation field rendering algorithm provided in an embodiment of the present application;

[0044] Figure 6 This is a comparison diagram of the motion trajectories of the vehicle-mounted acquisition equipment obtained from the degradation analysis and ablation experiment provided in the embodiment of the present application;

[0045] Figure 7 This is a schematic diagram comparing the three-axis positions of the vehicle-mounted acquisition device obtained from the degradation analysis and ablation experiment provided in the embodiment of the present application;

[0046] Figure 8 This is a comparison chart of the rendering effect of the present application scheme provided by the embodiment of the present application and the rendering effect of the prior art; (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP;

[0047] Figure 9 This is another comparison diagram of the rendering effect of the present application solution provided by the embodiment of the present application and the rendering effect of the prior art; (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP;

[0048] Figure 10 This is an architecture diagram of a three-dimensional building defect detection device based on multi-sensor fusion provided in an embodiment of the present application;

[0049] Figure 11 This is an architectural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] The term "and / or" in this application describes an association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " in this application indicates that the associated objects are in an "or" relationship, for example, A / B means A or B.

[0052] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] First, the technical terms involved in the embodiments of this application are introduced.

[0054] (1) Long linear structure

[0055] The extension direction of its long linear structure generally cannot provide geometric constraints of the structure and belongs to a structural degenerate environment; usually the extension direction of its long linear structure is called the degenerate direction, and the direction perpendicular to it is called the non-degenerate direction.

[0056] (2) Visual images

[0057] The visual images mentioned in this application are obtained through a depth camera, which includes information such as color and depth.

[0058] Next, the technical solutions provided in the embodiments of this application are introduced.

[0059] Figure 1 This is an architecture diagram of the vehicle-mounted acquisition device provided by the embodiment of the present application; Figure 1 As shown, it includes: a laser radar, a wheel speed meter, an IMU, an imaging device, a vehicle body and a wheel; wherein the wheel speed meter is arranged on the wheel and moves synchronously with the wheel; the laser radar, the IMU and the imaging device are all arranged on the vehicle body.

[0060] Optionally, two laser radars can be included, one installed horizontally and the other installed vertically. The above-mentioned laser radar can be a 16-line laser radar. During the detection process of the on-board acquisition equipment, the laser radar must be fully moved and flipped to ensure that each laser radar fully collects the complete point cloud of the calibration scene.

[0061] Optionally, the imaging device is a depth camera. The camera integrates a stereo image sensor, an infrared signal transmitter, and an RGB sensor, and can capture images and calculate differences between images, collect color data, illuminate objects, and collect depth data.

[0062] Further optionally, the above-mentioned lidar is used to observe point cloud data of the building to be inspected; the imaging device is used to obtain visual images of the building; the IMU is used to observe the position and speed of the vehicle-mounted acquisition device; and the wheel speed meter is used to observe the driving speed of the vehicle-mounted acquisition device.

[0063] In an optional scenario, the building to be detected is a long linear structure, taking a tunnel as an example. The above building may have the following characteristics:

[0064] (1) Global Navigation Satellite System (GNSS) denial: GNSS signals cannot be received in tunnels, and positioning and navigation cannot be achieved by GNSS.

[0065] (2) Structural degradation environment. Tunnels are all long linear structures with generally circular cross-sections and no obvious structural features. They cannot provide geometric constraints along the tunnel's forward direction and belong to a structural degradation environment.

[0066] (3) Weak lighting. There is no natural light in the tunnel. Even with internal electric lights, the tunnel is still quite dark.

[0067] (4) Weak texture. Tunnel linings are mostly made of reinforced concrete. There is usually no texture information in the tunnel lining, so highway tunnels belong to weak texture environments.

[0068] It is understood that the solution provided by this application is applicable to scenarios where the interior of long linear structures is inspected. The characteristics of this scenario are similar to those of a tunnel scenario. To simplify the description, the tunnel scenario may be mentioned in the following specific embodiments to specifically illustrate the embodiments of this application. Those skilled in the art should understand that the tunnel scenario is merely an example and does not limit the solution of this application in any way.

[0069] Figure 2 This is a flow chart of a method for detecting three-dimensional building defects based on multi-sensor fusion provided in an embodiment of the present application; Figure 2 As shown, the following steps are included:

[0070] Step S101, obtaining data from at least one laser radar, IMU, wheel speed meter and imaging device located on a vehicle-mounted acquisition device when inspecting a building to be inspected; the building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs inspection inside it.

[0071] Step S102, based on ESKF, the data observed by the lidar, IMU and wheel speed meter are fused to determine the speed and posture of the on-board acquisition device; wherein, the data of the IMU and wheel speed meter are fused once through forward propagation, and then the fused data are fused twice with the point cloud data observed by the lidar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the lidar and the historical map is less than a preset threshold, the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the lidar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure.

[0072] Those skilled in the art will understand that to obtain a three-dimensional point cloud model of the building to be measured, the point cloud data observed by the lidar and the speed and posture information of the lidar are required; in the above-mentioned long linear structure building environment, it is difficult to obtain the above-mentioned speed and posture information through traditional positioning methods, so it is necessary to integrate the lidar, IMU and wheel speed meter data to achieve it.

[0073] In this process, the above-mentioned backpropagation includes:

[0074] Calculate the state of the vehicle-mounted acquisition device based on the changes in the point cloud data observed by the lidar, and construct an observation equation by combining the calculated state and the state after forward propagation fusion; the state includes at least: posture;

[0075] Constructing a Jacobian matrix based on the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation;

[0076] The minimum eigenvalue of the observation matrix is ​​solved. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the pose of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, retaining only the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

[0077] Furthermore, for example, constructing the observation equation includes:

[0078] y=h(x t )+V;

[0079] Among them, y represents the actual measurement value obtained by solving the point cloud data based on lidar observation, h(x t ) represents the observation equation of the state quantity, x t Represents the state quantity after forward propagation prediction, and V represents Gaussian white noise;

[0080] The Jacobian matrix H constructed based on the partial derivative of the observation equation h with respect to the error state quantity is:

[0081] The above observation matrix is: H T H.

[0082] In a more specific embodiment, based on the ESKF framework, the original data are the speed information obtained by the wheel speed meter, the three-axis acceleration and three-axis angular velocity obtained by the IMU, and the point cloud data obtained by the lidar. By fusing the data of multiple sensors, the 18-dimensional state of the on-board acquisition equipment is estimated, which are six three-dimensional vectors: position, angle, speed, accelerometer bias, gyroscope bias, and gravitational acceleration g.

[0083] First, the IMU and wheel speedometer are integrated for forward propagation. This forward propagation process uses the IMU's three-axis acceleration and three-axis angular velocity data to recursively calculate the kinematic state of the vehicle's acquisition equipment, obtaining high-frequency pose information. The wheel speedometer speed information is then used to update the velocity value in the state variable. The backward propagation process uses the pose information from the forward propagation to dedistort the laser point cloud. The point cloud residual is then calculated through iterative nearest point short-cuts. The solution that minimizes the point-to-line-surface residual is obtained through iterative solution, and the Kalman gain is calculated to update the pose information in the state variable.

[0084] Specifically, as an example, the forward propagation of IMU and wheel speed meter includes the following two main parts:

[0085] Part 1: Integrating the IMU measurement values ​​allows for state recursion. Acceleration integration yields velocity, velocity integration yields position, and angular velocity integration yields angle. This allows for the corresponding kinematic discrete equations. The kinematic equations are then discretized and the error state propagated.

[0086] δx←f(x,δx,u m , i)=F(x,u m )·δx+F i ·i

[0087] Among them, f(x, δx, u m , i) represents the motion equation of ESKF; vector x represents the predicted state, δx represents the error state vector, u m represents the input vector, including acceleration and angular velocity measurements, i represents the random perturbation vector, including velocity, attitude, acceleration and angular velocity; F(x, u m ) represents f with respect to the error state δ x The Jacobian matrix, F i represents the Jacobian matrix of f with respect to the perturbation vector i.

[0088] The error covariance matrix of the prediction propagation part in the error state Kalman filter is obtained by linearizing the error state kinematics. The error covariance matrix propagation equation is as follows:

[0089]

[0090] Among them, P represents the covariance matrix of the state quantity, F x and F i is f with respect to the error state δ x and the Jacobian matrix of the perturbation vector i, Q i is the covariance matrix of the perturbation vector; ← is the assignment symbol.

[0091] Part 2: When the vehicle-mounted acquisition equipment is driving normally in the tunnel, the speed of the detection robot along the vehicle's forward direction can be obtained through the measurement value of the wheel speed meter. Only the speed along the vehicle's forward direction, when there is no bumping or sideslipping, will not produce lateral or vertical speed due to the kinematic constraints of the vehicle-mounted acquisition equipment. At this time, the vehicle's three-axis speed value can be obtained.

[0092] During state estimation, the IMU data is first integrated to determine the position, velocity, and rotation of the state variables. After the velocity measurement is obtained, the state variables are updated based on the velocity value. Because the velocity is observed, combined with the IMU's kinematic model, the inferred position and pose do not diverge rapidly.

[0093] As an example, the above backward propagation includes the following two main parts:

[0094] Part 1: The observation model can be written as follows:

[0095] y=h(x t )+V

[0096] Among them, h(x t ) is the observation equation, and V is Gaussian white noise.

[0097] Calculate the Kalman gain according to the observation equation, update the state, and propagate the error to obtain:

[0098] K=PH T (HPH T +V) -1

[0099]

[0100] P←(I-KH)P

[0101] Among them, K is the Kalman gain, P is the predicted state quantity, Represents the optimal estimated update value of the state quantity; Jacobian matrix

[0102] Update the forecast status as follows:

[0103]

[0104] Among them, P, v, q, a b 、ω b They represent the position, velocity, rotation, accelerometer bias, and gyroscope bias after the status update respectively. represents the optimal estimate of each component in the error state, and θ represents the rotation angle.

[0105] Specifically, the actual measurement value is obtained by solving the point cloud data observed by the lidar based on the Iterative Closest Point (ICP) algorithm. The key to the ICP algorithm is to solve the precise pose by continuously iteratively finding corresponding points in the source and target point clouds and minimizing the residual between them.

[0106] Assume that the current number of iterations of the iterative Kalman filter is k, and the point cloud dedistorted by backpropagation is converted to the world coordinate system:

[0107]

[0108] Wherein, the superscript G represents the world coordinate system; Indicates the expected coordinate value of the feature point after the pose to be solved is transformed into the world coordinate system; I T L Indicates the transformation from the radar laser coordinate system to the IMU coordinate system; Represents the coordinate value of the feature point in the k-th frame radar laser coordinate system; represents the pose to be solved; f j represents the jth feature point; I k represents the kth frame IMU; m represents the total number of feature points.

[0109] Specifically, Is the pose to be solved, the distance between the feature point and the matching feature (line or plane) to generate the residual. For edge feature points, the residual is the distance from the point to the line; for plane feature points, the residual is the distance from the point to the plane. Define G i is the normal vector of the plane or edge, G q j is a point on a plane or edge, then the residual Expressed as:

[0110]

[0111] Among them, the above residual It is the residual function of the lidar point cloud observation.

[0112] By solving the observation equation linearly, the observation equation of the extended Kalman filter is derived. Subsequently, multiple iterative optimizations are required to reduce the error caused by the linearized observation equation. It can be proved that the solution method of the iterative extended Kalman filter and the optimization method based on Newton's method are mathematically equivalent, and can solve the problem with an unbiased estimate.

[0113] During the iteration process, if the residual is less than the threshold, the iteration is considered to have converged, and the optimal state quantity is estimated to be:

[0114]

[0115] The state quantity X k Update and convert to Lie group form Each lidar point cloud in the k-th scanning frame The map is updated by projecting everything into the world coordinate system using the following formula.

[0116]

[0117] Through iteration, tightly coupled state estimation of the lidar and IMU is achieved. The position and posture of the tunnel detection machine and the zero bias of the IMU can be updated through lidar observation, achieving more accurate state estimation and global point cloud.

[0118] Part II: In the simultaneous localization and mapping problem, degradation refers to the phenomenon that the algorithm may perform poorly in an environment with insufficient constraints due to the instability and inaccuracy of state estimation. For example, insufficient observation information in a specific direction or dimension makes it difficult for the algorithm to accurately estimate the state from the observation data.

[0119] LiDAR requires spatial geometric structure information to iteratively solve the pose information. Since long linear structures lack structural features in their extension direction and belong to a degenerate environment, the state estimation along the long linear extension method is difficult to estimate due to the lack of constraints. It must be degenerated and the pose obtained is projected to the non-degenerate method. The components of the solution in the degenerate direction are discarded, and the components in the non-degenerate direction are retained.

[0120] There is an observation matrix in the solution process. T The degradation factor D can be calculated by performing eigendecomposition on H as follows:

[0121] D=λ min +1

[0122] Perform feature decomposition on the point cloud feature matrix, λ min H T The minimum eigenvalue of H. If the minimum eigenvalue is less than the threshold, the direction of the eigenvector corresponding to the eigenvalue corresponds to the direction of degradation.

[0123] Define the following matrix:

[0124]

[0125] Among them, V p represents the degradation direction vector; V u represents the non-degenerate direction vector; V f Represents all direction vectors obtained by the solution; n represents the total number of solution directions.

[0126] The corresponding reprojection components are:

[0127]

[0128] in, is an estimate of the true state, is the estimated value obtained after nonlinear optimization, For the optimal solution, x′ p represents the increment of the solution in the degenerate direction; x′ u represents the increment of the solution in the non-degenerate direction.

[0129] like Figure 4 As shown, the solution is decomposed into degenerate and non-degenerate directions, represented by orange and blue, respectively.

[0130] Retaining the solution in the non-degenerate direction, the final solution is:

[0131]

[0132] Where Δx u Indicates the optimal estimated value of the state quantity after updating.

[0133] Further optionally, during backward propagation, the point cloud data of the laser radar observing the extension direction of the long linear structure (that is, only the point cloud data of the laser radar in this direction needs to be fused twice to update the posture information) is fused twice with the data after the first fusion.

[0134] Step S103 , combining the speed, posture, and point cloud data observed by the lidar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system.

[0135] It can be understood that the speed and posture used in step S103 are the speed and posture of the laser radar, which can be obtained by transforming the speed and posture of the vehicle-mounted collection device according to the positional relationship between the laser radar and the vehicle-mounted collection device.

[0136] It should be noted that in order to make the three-dimensional point cloud model more accurate, this step combines the point cloud data observed by all lidars to construct the three-dimensional point cloud model.

[0137] In this application, the vehicle-mounted acquisition device chooses to use LiDAR as the main sensor to realize the positioning and mapping functions of the robot. However, there are many limitations in using only LiDAR for positioning and mapping, because it lacks the ability to detect the inertial state of the vehicle-mounted acquisition device itself, resulting in large deviations in the point cloud measurement results when the vehicle-mounted acquisition device moves at a fast speed or there are bumps. The IMU can measure the acceleration and angular velocity of the vehicle-mounted acquisition device, and can recursively infer the prior state based on the IMU data, dedistort the LiDAR point cloud and use it as the initial value for point cloud matching, which can improve the accuracy and robustness of positioning and mapping. The wheel speed meter can provide speed observations for the vehicle-mounted acquisition device, which is used to update the speed state quantity in the state estimation. When the tunnel point cloud is degraded, the wheel speed meter provides a state estimate in the degradation direction, making the positioning more robust. Therefore, this application will integrate IMU, LiDAR and wheel speed meter data to realize the positioning and mapping functions of the tunnel environment.

[0138] In step S104, a corresponding three-dimensional Gaussian function is generated based on the three-dimensional point cloud model, the three-dimensional Gaussian function in the world coordinate system is projected to the coordinate system of the imaging device in combination with the posture, and differentiable rasterization is performed to obtain a preliminary rendered image of the three-dimensional point cloud model; a loss function is constructed in combination with the visual image observed by the imaging device, the three-dimensional Gaussian function is optimized, and the rendered three-dimensional visual model is obtained iteratively.

[0139] It can be understood that the posture used in step S104 is the posture of the imaging device, which can be obtained by transforming the speed and posture of the vehicle-mounted acquisition device according to the positional relationship between the imaging device and the vehicle-mounted acquisition device.

[0140] In one example, a corresponding three-dimensional Gaussian function is generated according to a three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by the spherical harmonic function, and then the opacity is set for the ellipsoid.

[0141] In one example, the differentiable rasterization uses a tile-based rasterizer, the rasterizer including backpropagation to track the accumulated opacity value; constraining the opacity by a sigmoid activation function to ensure gradient smoothing;

[0142] The loss function is constructed by combining L1 loss and structural similarity index terms.

[0143] In a specific embodiment, the three-dimensional point cloud model constructed in step S103 and the depth image obtained in step S101 are used as input, and a 3D Gaussian function is used as a scene representation. Compared with traditional point clouds, the 3D Gaussian function provides more continuity and smoothness. The quality of the scene representation is improved by optimizing the properties of the 3D Gaussian function, iteratively optimizing the properties such as position, opacity, covariance and spherical harmonic coefficients of the 3D Gaussian function, and using an adaptive density control step to dynamically increase or decrease the number of 3D Gaussian functions. Through adaptive density control, the method is able to generate a compact and accurate scene representation. Finally, the present application uses a fast differentiable rasterization method, which utilizes the differentiability of the 3D Gaussian function and the ability to project onto a two-dimensional plane to efficiently mix anisotropic projections and opacity, ensuring the correct visibility order. In addition, by constructing a residual between the rendered image obtained by rasterization and the original image, fast and accurate backpropagation is achieved. For a specific algorithm flowchart, see Figure 5 shown.

[0144] The core steps in radiance field rendering based on 3D Gaussian functions are 3D Gaussian function parameter optimization and adaptive density control. Through an iterative loop, the algorithm continuously samples training views to obtain camera pose and image information. This information is then used to project points onto the image, generating a rasterized image. A loss function is then calculated between the generated image and the reference image to measure the difference. Backpropagation of the loss function and parameter updates are performed to adjust the position, covariance, color, and transparency properties of the point cloud, thereby improving image quality.

[0145] If the refinement iteration conditions are met, the following operations are performed:

[0146] (1) For each Gaussian distribution (represented by Gaussian mean, covariance, weight, and transparency), check whether the transparency is lower than the threshold or whether the mean and covariance are too large. If the conditions are met, prune and remove the Gaussian distribution.

[0147] (2) If the loss gradient with respect to the position exceeds the threshold, a densification operation is performed. If the covariance norm of the point exceeds the threshold, a super-reconstruction operation is performed, i.e., the Gaussian distribution is split. Otherwise, an under-reconstruction operation is performed, i.e., the Gaussian distribution is cloned.

[0148] (3) Finally, the iteration counter is incremented and the next cycle continues until the algorithm converges. Through iterative optimization and dynamic adjustment of Gaussian distribution properties, the algorithm can improve the accuracy and fidelity of image generation, thus playing an important role in scene rendering.

[0149] In order to enable high-quality iterative optimization of scene representation and to achieve high-quality new perspective synthesis based on a sparse point cloud model, this application uses a 3D Gaussian function as the basic spatial representation unit. This method has differentiable volume representation properties and is an unstructured explicit representation that can be rendered quickly. The point cloud is initialized as a set of 3D Gaussian functions to represent the point cloud as an ellipsoid. The position of the ellipsoid is represented by the mean of the 3D Gaussian function, the shape is represented by the covariance of the 3D Gaussian function, and the color is represented by the spherical harmonic function. The opacity of the ellipsoid is then set. These parameters can be used to create and optimize the radiation field through iterative optimization steps. The key to the efficiency of the method of this application is that this application uses a tile-based rasterizer, which allows alpha blending of anisotropic spots and maintains visibility order through fast sorting. In addition, the fast rasterizer of this application also includes fast backpropagation by tracking the accumulated alpha values ​​without limiting the number of Gaussian functions that can receive gradients.

[0150] The present invention models the geometry as a set of 3D Gaussian functions without normals, which are defined by the center point (mean) μ of the complete 3D covariance matrix Σ defined in world space. During the rasterization projection process of the present invention, this Gaussian function is multiplied by the opacity α.

[0151] To avoid the computational overhead of automatic differentiation during training, we explicitly derive gradients for all parameters. The easy-to-optimize anisotropic covariance representation we employ enables us to optimize the 3D Gaussian function to capture the geometric relationships of different shapes in the scene, resulting in an accurate scene representation.

[0152] Furthermore, during the parameter optimization process, the parameters of the 3D Gaussian functions are optimized, including position p, opacity α, and covariance Σ. Furthermore, the coefficients of the spherical harmonics representing the color of each Gaussian ellipsoid are optimized to ensure that the color appearance of the scene is correctly captured. During the parameter optimization process, the number of 3D Gaussian functions is adaptively controlled to better represent the scene.

[0153] During the optimization process, the rendering result is compared with the input training view in the data, and the residual is constructed and back-propagated. However, due to the ambiguity of the 3D to 2D projection, the ellipsoid may be generated incorrectly, so the optimization algorithm of this application needs to have the ability to generate, delete or move the ellipsoid to correct the position error. In the compact scene representation, the quality of the covariance parameters of the 3D Gaussian distribution is critical, because a small number of large anisotropic Gaussian distributions have better performance in large uniform areas. Stochastic gradient descent technology is used for optimization, making full use of the standard GPU acceleration framework, and custom CUDA kernels can be added to achieve fast rasterization of this application. This step is critical to the efficiency of the optimization because it is the main computational bottleneck in the optimization process.

[0154] This application chooses to use the sigmoid activation function to constrain the parameter α to the range [0-1) and ensure gradient smoothness, while using the exponential activation function to control the covariance scale. This application provides a suitable starting state for the model by estimating the initial covariance matrix as an isotropic Gaussian matrix with an axis length equal to the average distance to the three nearest points. The standard exponential decay scheduling technique is used to adjust the position parameters. The loss function designed by this application method combines the L1 loss and the structural similarity index (D-SSIM) term, and balances the two by weighing the parameter λ:

[0155]

[0156] We first obtain an initial sparse point set from simultaneous localization and mapping, and then optimize it by adaptively controlling the number and density of Gaussian points per unit volume. This allows us to transition from the initial sparse Gaussian set to a denser set that better represents the scene and has the correct parameters. After the initial optimization, we perform densification every 100 iterations and remove any Gaussian distributions that are essentially transparent, i.e., whose transmittance α is less than a set threshold.

[0157] The goal of adaptive Gaussian control is to fill in blank areas, particularly those lacking geometric features, known as under-reconstructed regions. This method is also applicable to areas that are largely covered by Gaussian distributions, known as over-reconstructed regions. Both under-reconstructed and over-reconstructed regions exhibit large view-space gradients, essentially due to incomplete reconstruction. During the optimization process, the number of Gaussian functions is adaptively controlled to improve the reconstruction of these regions.

[0158] This application uses the magnitude of the average view space position gradient exceeding a threshold to perform Gaussian function densification. In the experiment, this application set this threshold to 0.0002. For small Gaussian functions in under-reconstructed areas, the existing Gaussian function is copied to create a new Gaussian function, which is then shifted along the direction of the position gradient. For large Gaussian functions in over-reconstructed areas, they are decomposed into smaller Gaussian functions, and the two new Gaussian functions replace the original Gaussian function. The scale is reduced by a factor of φ = 1.6, which is the optimal parameter selected after multiple experiments.

[0159] This allows for more effective control of the density and distribution of Gaussian functions, thereby improving the quality and accuracy of scene reconstruction. In under-reconstructed areas, this application detects and addresses the need to increase the system's total volume and the number of Gaussians. In over-reconstructed areas, this application maintains the total volume unchanged but increases the number of Gaussians. The optimization process of this application can be affected by incorrectly constructed floating Gaussians close to the input camera, which can prevent the Gaussian density from increasing unreasonably. To effectively control the increase in the number of Gaussians, this application adopts a strategy of setting the alpha value to a value close to zero every 3000 iterations. Gaussians with high opacity are then added as needed, while those with transmittance below a threshold are removed. These Gaussians may shrink or expand and may have significant overlap with other Gaussians. However, by periodically removing Gaussians that are very large in world space or occupy a large area in view space, the overall number of Gaussians is well controlled. The Gaussians in this application's model are always maintained as primitives in Euclidean space, and unlike other conventional rendering methods, this method does not require spatial compression or projection of distant or large Gaussians.

[0160] In order to achieve fast overall rendering and sorting to support efficient transparency blending, this application designs a fast rasterization method based on a tile-based Gaussian ellipsoid, which is inspired by recent software rasterization methods and pre-sorts the entire image, avoiding the huge amount of computation caused by sorting each pixel. The fast rasterizer of this application can efficiently perform backpropagation of any number of mixed Gaussian functions, and only requires a constant memory for each pixel, and the additional memory consumption is low. The rasterization process of this application is fully differentiable, and based on 2D projection, anisotropic 3D Gaussian functions can be rendered.

[0161] The method of the present application first divides the image into 16×16 tiles, and then eliminates the 3D Gaussian functions based on the viewing cone and each tile. The method only retains the Gaussian functions with a confidence interval of 99% that intersects the viewing cone. According to the amount of overlap between the Gaussian function and the tile, a key is assigned to each Gaussian function during its instantiation, which combines the view space depth and the tile encoding. The Gaussian functions are sorted using these keys using a fast GPU Radix sort. In the absence of additional pixel-level sorting, transparency blending is performed based on the initial sorting. Therefore, in some configurations, the transparency blending of the present application may be approximate, but as the spots gradually approach the size of a single pixel, this approximation becomes negligible. The present application found that this method significantly improves training and rendering performance, and does not produce visible artifacts in converged scenes.

[0162] After sorting the Gaussian functions, the application generates a list for each tile, ensuring that the first and last entries in the depth-sorted entries intersect with the given tile. During rasterization, the application launches a thread block for each tile. Each thread block first loads the Gaussian function data packet into shared memory, and then for a given pixel, it traverses the list from front to back to accumulate color and opacity, thereby maximizing the parallelism of data loading, sharing, and processing. When a pixel reaches the target saturation, the corresponding thread stops. The threads in the tile are periodically queried, and when all pixels reach saturation, that is, the opacity approaches 1, the processing of the entire tile is terminated.

[0163] Furthermore, the above Figure 1 The method flow may further include step S105, performing defect detection on the building to be inspected based on the three-dimensional visual model of the building to be inspected.

[0164] Among them, the specific algorithm of the above-mentioned defect detection can be selected from existing mature defect detection algorithms, and can be more finely selected according to the type of building, which will not be described in detail here.

[0165] Specifically, for the degradation analysis mentioned in step S102, this application performs degradation analysis based on the structural degradation environment of the tunnel, conducts ablation experiments on the degradation analysis part, and draws trajectory comparisons and three-axis position curves including and excluding degradation analysis, respectively. Figure 6 and Figure 7 .from Figure 6 It can be seen from the figure that when the degradation analysis is not included, the algorithm trajectory is not closed, which means that when the degradation analysis is not performed, the algorithm cannot correctly estimate the state. Figure 7 As shown in the figure, when no degradation analysis is performed, a good estimate of the direction perpendicular to the tunnel (X direction) can be made, and the error between the method of the present application and that of the present invention is small. However, the error in the direction along the tunnel is large. This is because when no degradation analysis is performed, due to the degradation of the tunnel point cloud data, only lateral motion can be constrained, resulting in unobservable longitudinal motion. Incorrect longitudinal motion observation leads to incorrect position estimation in the forward direction. Therefore, it can be concluded that the degradation processing performed by the present application plays a key role in the algorithm's ability to achieve accurate positioning in the degraded environment of long linear structures.

[0166] Figure 8 and Figure 9 These are two comparison diagrams of the rendering effect of the present application scheme provided by the embodiment of the present application and the rendering effect of the prior art; in the two diagrams, (a) represents the true value, (b) represents the rendering effect of the present application, and (c) represents the rendering effect of InstantNGP; Figure 8As shown in the figure, in the rendering results of the building interior, the algorithm of this application can better adapt to the changes in light and dark inside the tunnel. The rectangular edge of the tunnel exit at the far end of the tunnel is not clear enough, but overall it has achieved better results. InstantNGP has a poor rendering effect inside the tunnel, producing a lot of black shadows and missing a lot of details. Figure 9 As shown, in the rendering results of the internal facilities of the building, both the algorithm of this application and InstantNGP achieved good rendering effects and were able to clearly render the internal facilities. However, the method of this application achieved clearer results in the facilities in the red frame area, and the rendering results of InstantNGP were slightly blurred.

[0167] In summary, the accuracy and clarity of the three-dimensional visual model rendered using the solution of the present application are significantly improved compared to the existing technology.

[0168] Figure 10 This is a diagram showing the architecture of a three-dimensional building defect detection device based on multi-sensor fusion provided in an embodiment of the present application; Figure 10 Shown, including:

[0169] The data acquisition unit 1010 is used to acquire data from the laser radar, IMU, wheel speed meter, and imaging device on the vehicle-mounted acquisition device when inspecting the building to be inspected; the building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted acquisition device performs inspections inside it;

[0170] The data fusion unit 1020 is used to fuse the data observed by the lidar, IMU, and wheel speed meter based on the ESKF to determine the speed and posture of the vehicle-mounted acquisition device; wherein, the IMU and wheel speed meter data are fused once through forward propagation, and then the fused data is secondary fused with the point cloud data observed by the lidar through backward propagation; during the backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the lidar and the historical map is less than a preset threshold, the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined based on the historical frame point cloud data observed by the lidar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure;

[0171] The point cloud model building unit 1030 is used to combine the speed, posture, and point cloud data observed by the lidar to obtain a three-dimensional point cloud model of the building to be detected in the world coordinate system;

[0172] The visual model building unit 1040 is used to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system to the coordinate system of the imaging device in combination with the posture, and perform differentiable rasterization to obtain an image of the three-dimensional point cloud model that is initially rendered; construct a loss function in combination with the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iteratively obtain the rendered three-dimensional visual model.

[0173] Further optionally, the data fusion unit 1020 solves the state quantity of the on-board acquisition device based on the changes in the point cloud data observed by the lidar during backward propagation, and constructs an observation equation by combining the solved state quantity and the state quantity after forward propagation fusion; the state quantity includes at least: posture; constructs a Jacobian matrix according to the partial derivatives of the observation equation relative to the state quantity after forward propagation fusion, and transposes the Jacobian matrix and multiplies it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solves the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

[0174] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program unit in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0175] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, such as Figure 11 As shown, the electronic device includes a processor 1111, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1111, the communication interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1111 can call the logic instructions in the memory 1130 to execute the method in the above embodiment.

[0176] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0177] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0178] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0179] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0180] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0182] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0183] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for detecting three-dimensional building defects based on multi-sensor fusion, characterized in that: include: Acquire data from at least one laser radar, an inertial measurement unit (IMU), a wheel speed meter, and an imaging device on a vehicle-mounted acquisition device when inspecting a building to be inspected; The building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted collection equipment performs inspections inside it; Based on the error state iterative extended Kalman filter (ESKF), the data observed by the lidar, IMU and wheel speed meter are fused to determine the speed and posture of the on-board acquisition device; wherein, the data of the IMU and wheel speed meter are fused once through forward propagation, and then the fused data are fused twice with the point cloud data observed by the lidar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the lidar and the historical map is less than a preset threshold, the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined according to the historical frame point cloud data observed by the lidar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure; Combining the speed, posture, and point cloud data observed by the lidar, a three-dimensional point cloud model of the building to be inspected in a world coordinate system is obtained; Generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system into the coordinate system of the imaging device based on the pose, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function based on the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iteratively obtain a rendered three-dimensional visual model; Also includes: Defect detection is performed on the building to be inspected based on its 3D visual model.

2. The method according to claim 1, characterized in that The back propagation includes: Calculate the state of the vehicle-mounted acquisition device based on the changes in the point cloud data observed by the lidar, and construct an observation equation by combining the calculated state and the state after forward propagation fusion; the state includes at least: posture; Constructing a Jacobian matrix based on the partial derivative of the observation equation with respect to the error state quantity, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain an observation matrix corresponding to the observation equation; The minimum eigenvalue of the observation matrix is ​​solved. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the pose of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, retaining only the pose solution in the non-degenerate direction; the degenerate direction is the extension direction of the long linear structure.

3. The method according to claim 1, characterized in that Generate a corresponding three-dimensional Gaussian function according to a three-dimensional point cloud model, including: initializing the three-dimensional point cloud model into a set of three-dimensional Gaussian functions, each three-dimensional Gaussian function is represented as an ellipsoid, the position of the ellipsoid is represented by the mean of the three-dimensional Gaussian function, the shape is represented by the covariance of the three-dimensional Gaussian function, the color is represented by the spherical harmonic function, and then setting the opacity of the ellipsoid.

4. The method according to claim 3, characterized in that The differentiable rasterization uses a tile-based rasterizer that includes backpropagation to track the accumulated opacity value; the opacity is constrained by a sigmoid activation function to ensure gradient smoothing; The loss function is constructed by combining L1 loss and structural similarity index terms.

5. The method according to claim 2, characterized in that The construction of the observation equation includes: ； in, Represents the actual measurement value obtained by solving the point cloud data based on lidar observations, The observation equation representing the state quantity, Represents the state quantity after forward propagation prediction, V represents Gaussian white noise; The Jacobian matrix H for: ; The observation matrix is: .

6. The method according to any one of claims 1 to 5, characterized in that The speed and posture of the laser radar and the imaging device are obtained by transforming the speed and posture of the vehicle-mounted acquisition device according to the positional relationship between the laser radar and the imaging device and the vehicle-mounted acquisition device; and / or During the backward propagation, the point cloud data of the laser radar that observes the extension direction of the long linear structure is fused with the data after the primary fusion for a second time.

7. A three-dimensional building defect detection device based on multi-sensor fusion, characterized in that: include: A data acquisition unit is used to acquire data from the laser radar, IMU, wheel speed meter and imaging device on the vehicle-mounted acquisition device when inspecting the building to be inspected; The building to be inspected is a long linear structure with a hollow interior, and the vehicle-mounted collection equipment performs inspections inside it; A data fusion unit is used to fuse the data observed by the lidar, IMU, and wheel speed meter based on the ESKF to determine the speed and posture of the on-board acquisition device; wherein, the data of the IMU and wheel speed meter are fused once through forward propagation, and then the fused data is fused twice with the point cloud data observed by the lidar through backward propagation; during backward propagation, if the minimum eigenvalue corresponding to the residual between the current frame point cloud data observed by the lidar and the historical map is less than a preset threshold, the posture of the on-board acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained for secondary fusion; the historical map is a point cloud set in the world coordinate system determined based on the historical frame point cloud data observed by the lidar, and the non-degenerate direction is perpendicular to the extension direction of the long linear structure; A point cloud model building unit is used to combine the speed, posture, and point cloud data observed by the lidar to obtain a three-dimensional point cloud model of the building to be inspected in the world coordinate system; A visual model building unit is configured to generate a corresponding three-dimensional Gaussian function based on the three-dimensional point cloud model, project the three-dimensional Gaussian function in the world coordinate system into the coordinate system of the imaging device in combination with the pose, and perform differentiable rasterization to obtain a preliminary rendered image of the three-dimensional point cloud model; construct a loss function based on the visual image observed by the imaging device, optimize the three-dimensional Gaussian function, and iteratively obtain a rendered three-dimensional visual model; Also includes: Defect detection is performed on the building to be inspected based on its 3D visual model.

8. The device according to claim 7, characterized in that The data fusion unit calculates the state quantity of the vehicle-mounted acquisition device based on the changes in the point cloud data observed by the lidar during backward propagation, and constructs an observation equation by combining the calculated state quantity with the state quantity fused by forward propagation; The state quantity includes at least: posture; constructing a Jacobian matrix based on the partial derivatives of the state quantity after forward propagation fusion of the observation equation, and transposing the Jacobian matrix and multiplying it by the Jacobian matrix to obtain the observation matrix corresponding to the observation equation; and solving the minimum eigenvalue of the observation matrix. If the minimum eigenvalue is less than a preset threshold, the eigenvector direction corresponding to the eigenvalue is used as the degenerate direction, and the posture of the vehicle-mounted acquisition device calculated based on the current frame point cloud data is decomposed, and only the posture solution in the non-degenerate direction is retained; the degenerate direction is the extension direction of the long linear structure.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

Images