Error suppression method based on unmanned aerial vehicle LiDAR modeling in complex scene

Abstract

The invention relates to the technical field of unmanned aerial vehicle LiDAR modeling, and particularly provides an unmanned aerial vehicle LiDAR modeling error suppression method based on a complex scene, and the method comprises the steps: collecting the point cloud data of the surface of a to-be-modeled target rod through an unmanned aerial vehicle, and the image data of a column device of the target rod; extracting geometric features of the point cloud data and textural features of the image data, and registering the geometric features and the textural features in combination with the pose data of the unmanned aerial vehicle; inputting the geometric features and the texture features into an improved deep learning network model, and fusing the geometric features and the texture features through the deep learning network model to obtain fused features; wherein the deep learning network model comprises a geometry-texture joint constraint and a target function. According to the error suppression method based on unmanned aerial vehicle LiDAR modeling in the complex scene, the registration capability of a three-dimensional model established for a complex target can be improved.

Description

Technical Field

[0001] This invention relates to the field of UAV LiDAR modeling technology, and in particular to an error suppression method for UAV LiDAR modeling in complex scenarios. Background Technology

[0002] In recent years, UAV LiDAR technology has become a core tool for modeling power equipment in complex scenarios, thanks to its non-contact, high-precision ranging capabilities, in the face of the challenges of inspecting complex, high-altitude pole-mounted equipment (such as switches, insulators, and circuit breakers). However, under complex conditions such as dynamic flight, building obstruction, and heterogeneous multi-source data, UAV LiDAR modeling faces the bottleneck problem of error propagation chain diffusion.

[0003] In existing technologies, error suppression strategies for UAV LiDAR (Light Detection and Ranging) modeling often focus on optimizing a single component. However, this method has the following main limitations:

[0004] First, existing studies often focus on isolated optimization of point cloud registration or data fusion processes, lacking a systematic analysis of the entire error propagation model from "UAV disturbance → LiDAR point cloud generation → multimodal registration → 3D reconstruction". For example, the mismatch between the extrinsic parameter drift of the UAV and the LiDAR scanning frequency during dynamic flight can lead to the accumulation of nonlinear errors, but existing ICP (Iterative Closest Point) algorithms do not consider such coupling effects.

[0005] Secondly, global registration of massive LiDAR point clouds relies on centralized computing. Existing BA (Bundle Adjustment) algorithms take more than 2 hours to process in a single-machine environment, which cannot meet the real-time requirements of power emergency inspection.

[0006] Therefore, traditional 3D modeling models are difficult to achieve ideal results in terms of modeling timeliness and registration capability when faced with complex targets. Summary of the Invention

[0007] The error suppression method for UAV LiDAR modeling in complex scenarios provided by the embodiments of the present invention at least solves the problem of insufficient registration capability of traditional 3D modeling models when facing complex targets.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] The first aspect of this invention provides an error suppression method for UAV LiDAR modeling in complex scenes, comprising the following steps: collecting point cloud data of the surface of a target pole to be modeled and image data of the pole-mounted equipment using a UAV; extracting geometric features from the point cloud data and texture features from the image data, and registering the geometric and texture features with the pose data of the UAV; inputting the geometric and texture features into an improved deep learning network model, and fusing the geometric and texture features through the deep learning network model to obtain fused features; wherein the deep learning network model includes: geometric-texture joint constraints and an objective function; the objective function is a function that dynamically solves for the minimum reprojection error when fusing the geometric and texture features; and constructing a three-dimensional real-world model of the target pole after error suppression based on the fused features.

[0010] Preferably, before extracting the geometric features of the point cloud data and the texture features of the image data, the method includes: distributing the point cloud data, image data, and pose data to various nodes of a distributed computing framework; the nodes include: a first node, a second node, and a third node; extracting the geometric features of the point cloud data through the first node; extracting the texture features of the image data through the second node; and registering the geometric features and texture features by combining them with the pose data of the UAV through the third node.

[0011] Preferably, the geometric features of the point cloud data and the texture features of the image data are extracted, and the geometric and texture features are registered in conjunction with the pose data of the UAV. This includes: receiving the point cloud data through a first node, performing statistical filtering, normal vector estimation, and missing data imputation on the point cloud data, and extracting geometric features; wherein multiple first nodes are provided, and each first node receives a portion of the point cloud data; receiving the image data through a second node, performing distortion correction, denoising, enhancement, and filtering on the image data, and extracting texture features; receiving the pose data through a third node, performing Kalman filtering on the pose data; and through the third node working in conjunction with the first and second nodes, performing temporal alignment and spatial registration of the point cloud data, image data, and pose data.

[0012] Preferably, the point cloud data, image data, and pose data are time-aligned by the third node working in collaboration with the first and second nodes, including: combining the pose data and aligning the point cloud data with the image data by comparing timestamps to obtain time-aligned point cloud temporal data and pixel temporal data; wherein the timestamps are recorded when the image data and the point cloud data are acquired.

[0013] Preferably, the point cloud data, image data, and pose data are spatially registered by the third node in collaboration with the first and second nodes. This includes: calibrating the positioning and attitude determination system with the LiDAR to obtain a first rotation matrix and translation vector; projecting the pose data onto the LiDAR coordinate system for a first registration based on the first rotation matrix and translation vector to obtain three-dimensional pose coordinates; obtaining a camera intrinsic parameter matrix; calibrating the LiDAR with the camera to obtain a second rotation matrix; and projecting the point cloud temporal data and three-dimensional pose coordinates onto a pixel coordinate system for a second registration based on the second rotation matrix and the camera intrinsic parameter matrix to obtain two-dimensional pose coordinates and two-dimensional point cloud coordinates spatially registered with the pixel coordinates in the image data.

[0014] Preferably, before inputting the geometric features and texture features into the improved deep learning network model, the method further includes: acquiring point cloud data of the surface of the target pole, image data of the equipment on the target pole, and recording the pose data of the UAV; extracting the geometric features of the point cloud data and the texture features of the image data, and registering the geometric features and texture features in conjunction with the pose data of the UAV; training the deep learning network model with the geometric features and texture features, combined with geometric-texture joint constraints and an objective function, to obtain the improved deep learning network model.

[0015] Preferably, before training the deep learning network model by combining the geometric features and the texture features with geometric-texture joint constraints and an objective function to obtain an improved deep learning network model, the method further includes: calculating the difference between the geometric attributes of the geometric features and the texture features to establish geometric consistency constraints; calculating the difference between the texture attributes of the texture features and the geometric features of the point cloud data to establish texture similarity constraints; and weighted summing the geometric consistency constraints and the texture similarity constraints to establish geometric-texture joint constraints.

[0016] Preferably, before training the deep learning network model by combining the geometric features and the texture features with geometric-texture joint constraints and an objective function to obtain an improved deep learning network model, the method further includes: dynamically calculating the reprojection error between the two-dimensional coordinates of the point cloud obtained by projecting the geometric features onto the pixel coordinate system of the texture features and the actual projected coordinates; minimizing the reprojection error to obtain a projection error term; using a robust kernel function to process the difference in distance between adjacent two-dimensional coordinates of the point cloud to remove the interference of outliers to obtain a robust kernel function term; and multiplying the projection error term by the robust kernel function term to establish the objective function.

[0017] Preferably, based on the fusion features, constructing a three-dimensional real-world model of the target pole after error suppression includes: solving the collinearity condition equation based on the fusion features to obtain the object coordinates of the target pole; wherein, the collinearity condition equation is established by combining the interior orientation elements of the camera carried by the UAV, describing the mapping relationship between the camera center of the UAV and the object point of the target pole; and constructing a three-dimensional real-world model of the target pole after error suppression based on the object coordinates and the geometric-texture information in the fusion features.

[0018] A second aspect of the present invention provides an electronic device, comprising: a processor, and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform an error suppression method based on LiDAR modeling of a UAV in a complex scene.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0020] This invention provides an error suppression method for UAV LiDAR modeling in complex scenes. By recording the UAV's pose data, it fuses the geometric features of a target pole with the texture features of a switch on a pole. The UAV's pose data is used to correct its extrinsic parameters, providing centimeter-level positioning for geometric features and sub-pixel-level matching for texture features. This solves the problem of extrinsic parameter drift and LiDAR scanning mismatch. Furthermore, this invention uses a deep network model to fuse geometric and texture features, improving fusion efficiency. Optimizing the fused features through an objective function and joint geometry-texture constraints dynamically corrects reprojection errors generated during the fusion of geometric and texture features, thereby improving the registration accuracy of feature fusion and enhancing the registration capability of LiDAR modeling. This method can adapt to the modeling needs of complex target scenes. Attached Figure Description

[0021] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an error suppression method based on UAV LiDAR modeling in complex scenarios, according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of the electronic device created by this invention. Detailed Implementation

[0024] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] In related technologies, for example, patent CN119251424A discloses a method for generating high-precision 3D models from building point cloud data. This method optimizes point cloud registration, completion, segmentation, and parameter extraction steps through deep learning, providing an efficient method for automated generation of BIM models from point cloud data. Patent CN114638867A discloses a point cloud registration method and system based on a feature extraction module and dual quaternions, fully mining global and local information in the point cloud and effectively compensating for the lack of local features in the global feature extraction stage of point cloud registration.

[0026] These methods demonstrate good registration accuracy in static scenarios, but face a dual challenge under dynamic UAV flight conditions: First, the time-varying characteristics of sensor extrinsic parameters and the spatiotemporal asynchronous effect generated by low-frequency LiDAR sampling (Δt≥50ms) can lead to a compound error accumulation that exceeds the linear growth pattern; Second, the computational complexity of traditional BA algorithms based on centralized computing architectures is so large that the optimization time on a single machine exceeds 2 hours (measured on Intel Xeon Gold 6248R), which is difficult to meet the stringent requirements of real-time response (<5 minutes) for power emergency inspection.

[0027] like Figure 1 As shown, in order to improve the registration capability of 3D models built for complex targets, the embodiments of this invention provide an error suppression method for UAV LiDAR modeling in complex scenes, including the following steps:

[0028] Step S1: Collect point cloud data of the surface of the target pole to be modeled, as well as image data of the equipment on the target pole, using a drone.

[0029] Step S2: Extract the geometric features of the point cloud data and the texture features of the image data, and register the geometric features and texture features in conjunction with the pose data of the UAV.

[0030] Step S3: Input the geometric features and texture features into the improved deep learning network model, and fuse the geometric features and texture features through the deep learning network model to obtain the fused features; wherein, the deep learning network model includes: geometric-texture joint constraints and objective function; the objective function is a function that dynamically solves the minimum value of the reprojection error when fusing geometric features and texture features.

[0031] Step S4: Based on the fusion features, construct a 3D real-world model of the target pole after error suppression.

[0032] The trend of pole-mounted equipment becoming more complex and higher is significant. Due to long-term exposure to wind and rain erosion and bird pecking, the connection points are prone to metal exposure defects caused by insulation damage. It is necessary to rely on UAV LiDAR technology to model complex scenarios.

[0033] The embodiments of the present invention are mainly aimed at the inspection of 10kV pole-mounted equipment, which includes: pole-mounted switches, insulators, and pole-mounted circuit breakers.

[0034] UAV LiDAR technology is a remote sensing technology that integrates UAV platforms with LiDAR (Light Detection and Ranging). By using UAVs equipped with LiDAR sensors, it enables high-precision three-dimensional spatial information acquisition and modeling of the ground or target objects.

[0035] In step S1, the point cloud data is obtained by scanning the surface of the target pole using the UAV's LiDAR sensor. Specifically, the LiDAR acquires the three-dimensional point cloud data of the target pole with a certain scanning frequency, field of view, and dynamically adjusted pulse-echo mode, with a point density of not less than 50 points / square meter, and is capable of collecting point cloud data of the irregular surface of the target pole.

[0036] The image data was acquired by using a camera carried by a drone to capture images of the switches on the target pillar. Specifically, the drone was equipped with a full-frame camera and captured high-resolution images according to aerial photography specifications of 80% forward overlap and 70% lateral overlap.

[0037] The positioning and orientation data is obtained by real-time transmission and recording through the Position and Orientation System (POS) on the UAV.

[0038] The POS system includes a combination of Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS), and the preferred pose data of the UAV includes the UAV's position information and attitude angles.

[0039] This invention provides an error suppression method for UAV LiDAR modeling in complex scenes. By recording the UAV's pose data, it fuses the geometric features of a target pole with the texture features of a switch on a pole. The UAV's pose data is used to correct its extrinsic parameters, providing centimeter-level positioning for geometric features and sub-pixel-level matching for texture features. This solves the problem of extrinsic parameter drift and LiDAR scanning mismatch. Furthermore, this invention uses a deep network model to fuse geometric and texture features, improving fusion efficiency. Optimizing the fused features through an objective function and joint geometry-texture constraints dynamically corrects reprojection errors generated during the fusion of geometric and texture features, thereby improving the registration accuracy of feature fusion and enhancing the registration capability of LiDAR modeling. This method can adapt to the modeling needs of complex target scenes.

[0040] Furthermore, before extracting the geometric features of the point cloud data and the texture features of the image data in step S2, the method of this embodiment of the invention further includes the following steps:

[0041] Step S021: Distribute the point cloud data, image data, and pose data to the nodes of the distributed computing framework; the nodes include: the first node, the second node, and the third node.

[0042] Step S022: Extract the geometric features of the point cloud data through the first node.

[0043] Step S023: Extract texture features from image data through the second node.

[0044] Step S024: By combining the pose data of the UAV with the third node, the geometric features and texture features are registered.

[0045] This invention distributes point cloud data, image data, and pose data to different nodes in a distributed computing framework. By having multiple nodes process different tasks in parallel, algorithms and resource configurations can be optimized for different data characteristics, breaking through the computing power bottleneck of a single computing node, improving overall processing efficiency, and meeting the needs of real-time processing of multi-source data.

[0046] Furthermore, this invention combines the pose data of the UAV to register geometric and texture features, which can solve the problems of time asynchrony between sensors (such as the difference in sampling time between the camera and LiDAR) and spatial extrinsic errors (such as sensor installation position deviation), achieve precise integration of geometric and texture features, and eliminate point cloud drift caused by UAV flight vibration and wind speed disturbance.

[0047] Furthermore, step S2 preferably includes:

[0048] Step S21: Receive point cloud data through the first node, perform statistical filtering, normal vector estimation, and missing data imputation on the point cloud data, and extract geometric features; wherein, multiple first nodes are set, and each first node receives a portion of the point cloud data;

[0049] Step S22: Receive image data through the second node, perform distortion correction, denoising, enhancement and filtering on the image data, and extract texture features;

[0050] Step S23: Receive pose data through the third node and perform Kalman filtering on the pose data;

[0051] Step S24: Through the collaborative work of the third node, the first node, and the second node, the point cloud data, image data, and pose data are time-aligned and spatially registered.

[0052] Specifically, the point cloud data collected by the LiDAR sensor in this embodiment of the invention is massive and requires intensive computation. By distributing the point cloud data to multiple first nodes for parallel processing, computational efficiency can be improved.

[0053] Furthermore, the point cloud data is segmented and distributed to multiple first nodes. The distribution of point cloud data can be dynamically adjusted based on the current computing power of each first node to meet computational demands.

[0054] Preferably, each first node uses a statistical outlier removal algorithm to filter out anomalies, which can effectively filter out random noise points and isolated points caused by mirror reflection and environmental interference from the lidar, making the point cloud distribution closer to the real scene structure.

[0055] Furthermore, the RANSAC algorithm is used to segment the building's walls, floors, and other planes, extracting feature elements from the wall point cloud. For the segmented wall point cloud, a surface mesh is generated using Delaunay triangulation. The mesh boundaries are extracted as the initial contour, and RANSAC line / curve fitting is used to simplify the contour, identifying regular shapes such as rectangles and polygons to extract the building's planar outline. The Harris 3D corner detector or ISS (Integral Shape Signatures) keypoint detection is used to detect the intersection points (3D corners) of window frame edges. Combined with point cloud normal vector analysis (such as the angle between the window frame plane and the wall plane), corner points in the window frame area are selected. By performing small-neighborhood plane fitting on the wall point cloud, the normal vectors of each point are calculated, and the brick joint boundary lines can be detected through the mutation of the normal vectors.

[0056] By preprocessing the point cloud data through the first node, the geometric features that can be extracted include: building plan outline, window frame area corner points, and brick joint boundary lines.

[0057] In step S22, after the second node receives the image data, it first preprocesses the image data and then extracts the texture features from the image data.

[0058] Specifically, image data preprocessing includes:

[0059] First, based on the camera's intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix, translation vector) obtained from camera calibration, lens distortion is corrected using a polynomial model to eliminate barrel / pincushion distortion.

[0060] Next, nonlocal means are used to filter the pixel coordinates to eliminate Gaussian noise and preserve details.

[0061] Finally, based on the filtered pixel coordinates, a histogram is generated, and then an equalization transformation is applied to expand the dynamic range of the image, improve image contrast, and solve the problem of texture loss in backlit or overexposed areas.

[0062] Extracting texture features from image data includes:

[0063] Keypoints are detected by calculating the difference of Gaussian (DoG) for each pixel in the image. Then, feature descriptors are generated based on the gradient directions of the pixels around the keypoints, and the pixel coordinates of each pixel in the image are obtained.

[0064] Key points are local feature points that are scale-invariant, rotation-invariant, and illumination-invariant, including corner points, edges, and spots in the image.

[0065] Therefore, the texture features extracted in step S22 include: key points, feature descriptors, and pixel coordinates.

[0066] In step S23, Kalman filtering can eliminate high-frequency noise in pose data, improve the stability and accuracy of pose estimation, and provide more reliable transformation parameters for spatial registration.

[0067] Furthermore, step S24 preferably includes the following steps:

[0068] Step S241: Combine the pose data and align the point cloud data with the image data by comparing the timestamps to obtain time-aligned point cloud temporal data and pixel temporal data; wherein, the timestamps are recorded when acquiring image data and point cloud data.

[0069] Step S242: By calibrating the external parameters of the positioning and attitude determination system and the lidar, the first rotation matrix and translation vector are obtained. Based on the first rotation matrix and translation vector, the pose data is projected onto the lidar coordinate system for the first registration to obtain the three-dimensional pose coordinates.

[0070] Step S243: Obtain the camera intrinsic parameter matrix, and obtain the second rotation matrix by calibrating the lidar and the camera extrinsic parameters. Based on the second rotation matrix and the camera intrinsic parameter matrix, project the point cloud temporal data and pose 3D coordinates onto the pixel coordinate system for a second registration, and obtain the pose 2D coordinates and point cloud 2D coordinates registered with the pixel coordinate space in the image data.

[0071] Specifically, in step S243, the embodiment of the present invention completes the extrinsic parameter calibration of the lidar and camera through the degree-of-freedom calibration target, and calculates the second rotation matrix and the second translation vector using singular value decomposition.

[0072] Furthermore, after obtaining the two-dimensional coordinates of the point cloud, the method of this embodiment of the invention further includes:

[0073] Step S244: The method of minimizing reprojection error is used to improve the registration accuracy of the second registration.

[0074] Specifically, the function used to minimize the reprojection error can be expressed as follows:

[0075]

[0076] in, This represents the two-dimensional coordinates of the point cloud obtained by projecting the point cloud coordinates of the i-th feature point onto the pixel coordinates. R represents the point cloud coordinates of the i-th feature point acquired by the LiDAR sensor; R represents the second rotation matrix; T represents the second translation vector; and π represents the camera projection function.

[0077] By performing the above processing and registration on point cloud data, image data, and pose data, this invention can improve the accuracy of data acquisition, thereby facilitating the detailed modeling of complex target pole structures.

[0078] Furthermore, prior to step S3, the method of this embodiment of the invention further includes:

[0079] Step S031: Collect point cloud data of the surface of the target pole, image data of the equipment on the target pole, and record the pose data of the UAV;

[0080] Step S032: Extract the geometric features of the point cloud data and the texture features of the image data, and register the geometric features and texture features in conjunction with the pose data of the UAV.

[0081] Step S033: Combine geometric features and texture features with geometric-texture joint constraints and objective function to train the deep learning network model, thereby obtaining an improved deep learning network model.

[0082] Specifically, the point cloud data of the target pole is acquired by the LiDAR sensor of the UAV, the image data of the equipment on each column of the target pole is acquired by the camera on the UAV, and the pose data of the UAV is obtained by the UAV's POS system.

[0083] Step S032 is preferably the same as step S2, and will not be repeated here.

[0084] Before step S033, the method of this embodiment of the invention further includes:

[0085] Step S0331: Calculate the difference between the geometric attributes of geometric features and texture features, and establish geometric consistency constraints.

[0086] Step S0332: Calculate the difference in texture attributes between texture features and geometric features of point cloud data, and establish texture similarity constraints.

[0087] Step S0333: Weighted summation of geometric consistency constraints and texture similarity constraints to establish geometric-texture joint constraints.

[0088] Specifically, the geometric consistency constraint established in step S0331 can preferably be expressed as follows:

[0089]

[0090] in, Represents geometric consistency constraints; Let j be the j-th geometric feature point of the lidar; This represents the j-th geometric feature point of the image.

[0091] Specifically, Preferably, it is the normal vector of the j-th point cloud coordinates; The geometric information of the feature points corresponding to the LiDAR is selected from the pixel coordinates.

[0092] The texture similarity constraint established in step S0332 can preferably be expressed as follows:

[0093]

[0094] in, Represents geometric consistency constraints; This represents the k-th texture feature point of the LiDAR; This represents the k-th texture feature point of the image.

[0095] In step S0333, the established geometry-texture joint constraint can be expressed as the following expression:

[0096]

[0097] Where E represents the established geometry-texture joint constraint, i.e., the joint energy function; The weights representing geometric similarity constraints, This represents the weight of the texture similarity constraint.

[0098] Step S0334: Dynamically calculate the reprojection error between the two-dimensional coordinates of the point cloud obtained by projecting the geometric features onto the pixel coordinate system of the texture features and the actual projection coordinates; minimize the reprojection error to obtain the projection error term.

[0099] Step S0335: Use a robust kernel function to process the difference in distance between the two-dimensional coordinates of adjacent point clouds, remove the interference of outliers, and obtain the robust kernel function term;

[0100] Step S0336: Multiply the projection error term by the robust kernel function term to establish the objective function.

[0101] In step S0332, the texture similarity constraint is determined from the perspective of histogram intersection operation based on local image patches and gradient direction consistency analysis. It can be represented as the histogram of the k-th local image patch of point cloud data. By performing an intersection operation with the histogram of the corresponding patch of the camera image, the gradient direction distribution information within the local image patch is extracted. This represents the histogram of a local image patch at the k-th texture feature point.

[0102] In step S0336, the objective function is preferably a BA objective function established based on the bundle adjustment method, which can be expressed as follows:

[0103]

[0104] in, Indicates the projection error term. The i-th frame of UAV pose data includes: , This represents the two-dimensional coordinates of the point cloud obtained by projecting the point cloud coordinates onto the pixel coordinate system. Represents the coordinates of the j-th pixel, including: N and M represent natural numbers.

[0105] Denotes the robust kernel function term, where Represents a robust kernel function. Indicates and Different pixel coordinates, for The coordinates of adjacent pixels.

[0106] In this embodiment of the invention, geometric features and texture features are combined with geometric-texture joint constraints and objective functions to train a convolutional neural network, thereby obtaining a deep learning network model.

[0107] After obtaining the deep network learning model, the model is transferred to special scenarios such as power equipment and irregularly shaped buildings through transfer learning. The model is fine-tuned with a small number of labeled samples, so that the deep network learning model in this embodiment of the invention can be applied to various scenarios.

[0108] In step S3, geometric and texture features are input into a deep learning network model. The deep learning network model automatically captures the correspondence between geometric and texture features, fuses the geometric and texture features, captures the implicit correlation between point clouds and images, and optimizes the fused features through an objective function and joint geometry-texture constraints to achieve sub-pixel-level multi-source data alignment. Then, through semantic segmentation, deep semantic information can be obtained, including: brick joint direction and material reflection characteristics.

[0109] Furthermore, in step S4, based on the fused features, a 3D real-world model of the target pole after error suppression is constructed, including:

[0110] Step S4.1: Based on the fusion features, solve the collinearity condition equation to obtain the object coordinates of the target pole; wherein, the collinearity condition equation is established by combining the interior orientation elements of the camera carried by the UAV, and describes the collinearity condition equation between the camera center of the UAV and the object point of the target pole.

[0111] Step S4.2: Based on the object coordinates and combined with the geometric-texture information in the fused features, construct a 3D real-world model of the target pole after error suppression.

[0112] Specifically, the collinearity condition equation in step S4.1 can be expressed as follows:

[0113]

[0114]

[0115] in, (x, y) represents the focal length in the camera's interior orientation element; (x, y) represents the coordinates of the corresponding image point, i.e., the two-dimensional coordinates of the point cloud; (Xs, Ys, Zs) represents the coordinates of the shooting point, i.e., the spatial position of the drone's camera during shooting. , i=1,2,3, represent the exterior orientation element rotation matrix, which is a matrix composed of the cosine values ​​of the nine directions of the rotation matrix calculated by using trigonometric functions based on the three exterior orientation elements (heading angle, side tilt angle, image rotation angle); (X, Y, Z) represent the object coordinates to be calculated.

[0116] Based on object coordinates and fusion features, Poisson reconstruction is driven to generate topologically coherent triangular meshes. Based on multi-view image visibility analysis and GraphCut texture fusion, a high-fidelity 3D model of the scene is finally output, achieving both geometric accuracy (RMSE <2cm) and texture realism (SSIM >0.85).

[0117] Another embodiment of the present invention provides an electronic device, including: a processor, and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the error suppression method based on UAV LiDAR modeling in complex scenarios according to the embodiments of the present invention.

[0118] like Figure 2 As shown, the electronic device includes a computing unit 101, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 102 or a computer program loaded from a storage unit 108 into a random access memory (RAM) 103. The RAM 103 may also store various programs and data required for the operation of the electronic device. The computing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0119] Multiple components in the electronic device are connected to I / O interface 105, including: input unit 106, output unit 107, storage unit 108, and communication unit 109. Input unit 106 can be any type of device capable of inputting information into the electronic device. Input unit 106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 108 may include, but is not limited to, disks and optical discs. Communication unit 109 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0120] The computing unit 101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as computer programs tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 102 and / or communication unit 109. In some embodiments, the computing unit 101 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0121] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of embodiments of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0124] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0125] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0126] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0127] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An error suppression method based on UAV LiDAR modeling in complex scenarios, characterized in that, Includes the following steps: The point cloud data of the surface of the target pole to be modeled, as well as the image data of the pole-mounted equipment, are collected by drone. Extract the geometric features of the point cloud data and the texture features of the image data, and combine them with the pose data of the UAV to register the geometric features and texture features. The geometric and texture features are input into an improved deep learning network model, which then fuses the geometric and texture features to obtain fused features. The deep learning network model includes a geometric-texture joint constraint and an objective function. The objective function is a function that dynamically solves for the minimum reprojection error when fusing the geometric and texture features. Based on the fusion features, a three-dimensional real-world model of the target pole after error suppression is constructed.

2. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 1, characterized in that, Before extracting the geometric features of the point cloud data and the texture features of the image data, the method includes: The point cloud data, image data, and pose data are distributed to various nodes of the distributed computing framework; the nodes include: a first node, a second node, and a third node; Geometric features of the point cloud data are extracted using the first node; The texture features of the image data are extracted using the second node; The geometric and texture features are registered by combining the pose data of the UAV with the third node.

3. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 2, characterized in that, Extract geometric features from point cloud data and texture features from image data, and combine them with UAV pose data to register the geometric and texture features, including: The point cloud data is received by the first node, and statistical filtering, normal vector estimation and missing data imputation are performed on the point cloud data to extract geometric features; wherein, multiple first nodes are set, and each first node receives a portion of the point cloud data. The image data is received through the second node, and the image data is subjected to distortion correction, denoising, enhancement and filtering processes to extract texture features. The pose data is received through a third node, and the pose data is processed by Kalman filtering. By having the third node work in collaboration with the first and second nodes, the point cloud data, image data, and pose data are time-aligned and spatially registered.

4. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 3, characterized in that, By having a third node work in conjunction with the first and second nodes, the point cloud data, image data, and pose data are time-aligned, including: By combining the pose data, the point cloud data and image data are aligned by comparing timestamps to obtain time-aligned point cloud temporal data and pixel temporal data; wherein, the timestamps are recorded when the image data and the point cloud data are acquired.

5. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 3, characterized in that, By having the third node work in collaboration with the first and second nodes, spatial registration is performed on the point cloud data, image data, and pose data, including: By calibrating the positioning and attitude determination system with the lidar, the first rotation matrix and translation vector are obtained. Based on the first rotation matrix and translation vector, the pose data is projected onto the lidar coordinate system for the first registration, and the three-dimensional pose coordinates are obtained. The camera intrinsic parameter matrix is ​​obtained, and the second rotation matrix is ​​obtained by calibrating the lidar and the camera extrinsic parameters. Based on the second rotation matrix and the camera intrinsic parameter matrix, the point cloud temporal data and pose 3D coordinates are projected onto the pixel coordinate system for a second registration, resulting in pose 2D coordinates and point cloud 2D coordinates registered with the pixel coordinate space in the image data.

6. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 1, characterized in that, Before inputting the geometric and texture features into the improved deep learning network model, the method further includes: Collect point cloud data of the surface of the target pole, image data of the equipment on the target pole, and record the pose data of the UAV; Extract the geometric features of the point cloud data and the texture features of the image data, and combine them with the pose data of the UAV to register the geometric features and the texture features. The geometric features and texture features are combined with geometric-texture joint constraints and an objective function to train a deep learning network model, resulting in an improved deep learning network model.

7. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 6, characterized in that, Before training the deep learning network model by combining the geometric features and the texture features with geometric-texture joint constraints and an objective function to obtain an improved deep learning network model, the method further includes: Calculate the differences between the geometric attributes of the geometric features and the texture features, and establish geometric consistency constraints; Calculate the difference in texture attributes between the texture features and the geometric features of the point cloud data, and establish texture similarity constraints; The geometric consistency constraint and the texture similarity constraint are weighted and summed to establish a geometric-texture joint constraint.

8. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 6, characterized in that, Before training the deep learning network model by combining the geometric features and the texture features with geometric-texture joint constraints and an objective function to obtain an improved deep learning network model, the method further includes: The reprojection error between the two-dimensional coordinates of the point cloud obtained by projecting the geometric features onto the pixel coordinate system of the texture features and the actual projection coordinates is dynamically calculated. The reprojection error is minimized to obtain the projection error term. The robust kernel function is used to process the difference in distance between adjacent point cloud two-dimensional coordinates to remove the interference of outliers and obtain the robust kernel function term; The objective function is established by multiplying the projection error term by the robust kernel function term.

9. The error suppression method for UAV LiDAR modeling in complex scenarios according to claim 1, characterized in that, Based on the fusion features, a 3D real-world model of the target pole after error suppression is constructed, including: Based on the fusion features, the collinearity condition equation is solved to obtain the object coordinates of the target pole; wherein, the collinearity condition equation is established by combining the interior orientation elements of the camera carried by the UAV, and describes the collinearity condition equation describing the mapping relationship between the camera center of the UAV and the object point of the target pole. Based on the object coordinates and combined with the geometric-texture information in the fused features, a three-dimensional real-world model of the target pole after error suppression is constructed.

10. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform an error suppression method for UAV LiDAR modeling in complex scenarios according to any one of claims 1-9.

Images