Multi-modal semantic consistency gaussian mapping and positioning method for bridge unmanned aerial vehicle inspection

Abstract

This invention proposes a multimodal semantically consistent Gaussian mapping and localization method for bridge unmanned aerial vehicle (UAV) inspection. By tightly coupling visual and LiDAR data and introducing semantic consistency constraints, this method achieves high-precision localization and detailed 3D mapping of UAVs in weakly textured environments, providing a reliable technical solution for unmanned bridge inspection and intelligent operation and maintenance in civil engineering.

Description

Technical Field

[0001] This invention relates to the fields of bridge structural health monitoring, autonomous UAV inspection, 3D reconstruction and localization, computer vision and multimodal SLAM (Simultaneous Localization and Mapping), and can also be applied to intelligent operation and maintenance of civil infrastructure, autonomous navigation of unmanned systems and robot simultaneous localization and mapping. In particular, it relates to a multimodal semantically consistent Gaussian mapping and localization method for UAV bridge inspection. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) is a technology that enables robots to simultaneously localize and map in unknown environments. Its development has progressed from early methods based on Extended Kalman Filter (EKF) and Bayesian filtering, to mainstream frameworks represented by graph optimization and visual SLAM, and now to intelligent, robust systems integrating multiple sensors and deep learning. SLAM technology has been widely applied in civil infrastructure fields such as bridge inspection, especially in environments where GPS signals are obstructed, such as under beams. It can provide UAVs with autonomous localization, simultaneous mapping, and inspection path planning, while also enabling damage visualization. Currently popular SLAM methods mainly include the following two categories: (1) NeRF-based SLAM method NeRF-based methods implicitly model scenes using multilayer perceptrons (MLPs), generating high-fidelity visual effects and realistic lighting, suitable for high-precision map reconstruction and camera tracking. These methods typically achieve camera pose estimation and scene reconstruction by jointly optimizing photometric and geometric errors, providing high reconstruction accuracy and visual realism in static environments. However, due to the reliance on the MLP network for implicit representation, computational overhead is high, and real-time performance is insufficient, easily leading to performance bottlenecks in large-scale scenes or high-precision reconstruction. Furthermore, these methods rely on static scene assumptions, exhibiting poor adaptability to dynamic objects and environmental changes, making them unsuitable for complex applications such as bridge inspection. Some improved methods introduce dynamic object segmentation and motion estimation mechanisms to enhance robustness, but still suffer from low computational efficiency and limited dynamic adaptability.

[0003] (2) SLAM method based on 3D Gaussian splashing 3D Gaussian splashing methods improve rendering efficiency and large-scale scene processing capabilities by representing the scene as a 3D Gaussian volume. These methods typically combine geometric constraints and opacity optimization to achieve efficient and continuous scene reconstruction and camera pose tracking. Compared to NeRF methods, 3DGS methods have significant advantages in rendering speed and map-building efficiency, supporting real-time operations on larger scenes. However, their performance still relies on high-quality depth data, resulting in significant memory consumption; and the system exhibits poor scale adaptability in monocular settings or environments with significant scale variations. While multimodal fusion can improve robustness and localization accuracy, it suffers from high computational complexity, and a trade-off remains between real-time performance and mapping accuracy.

[0004] In summary, existing bridge inspection SLAM technologies still face numerous technical bottlenecks in complex structures, weak textures, and dynamic environments, making it difficult to simultaneously achieve real-time, multimodal, semantically consistent, and high-precision 3D mapping and localization. To address these issues, this invention proposes a multimodal, semantically consistent Gaussian SLAM 3D reconstruction and localization method for complex bridge structures. By tightly coupling visual and LiDAR data and introducing semantic consistency constraints, this method achieves high-precision localization and detailed 3D mapping for UAVs in weak-texture environments, providing a reliable technical solution for bridge inspection and unmanned inspection and intelligent operation and maintenance in civil engineering. Summary of the Invention

[0005] The purpose of this invention is to address the problems of poor real-time performance, insufficient multimodal fusion, and limited 3D reconstruction and positioning accuracy in existing bridge inspection SLAM technologies, particularly in complex structures and weakly textured environments. This invention proposes a multimodal semantically consistent Gaussian mapping and positioning method for bridge inspection using unmanned aerial vehicles (UAVs). Under complex bridge structural environments with limited GNSS signals and weak textures, this method achieves real-time 3D mapping and high-precision positioning based on multimodal (visual and LiDAR) semantic consistency Gaussian SLAM. This invention can be widely applied to unmanned inspection and high-precision 3D modeling of civil engineering structures such as bridges, buildings, and tunnels, and is also suitable for autonomous navigation and intelligent operation and maintenance scenarios of related unmanned systems.

[0006] This invention is achieved through the following technical solution: This invention proposes a multimodal semantically consistent Gaussian mapping and localization method for bridge UAV inspection, the method comprising the following steps: Step 1: Initialize the front-end pose using LiDAR point cloud and construct initial scene geometric constraints to provide prior pose for stable positioning of the UAV in weak texture, repetitive structure and GNSS denied environments; Step 2: Perform local neighborhood analysis on the lidar point cloud, estimate the surface normal vector of each point, and characterize the consistency between the normal vector and the principal axis direction of the corresponding 3D Gaussian volume; further, combine the local Gaussian distribution density and normal consistency to construct a weighted matching mechanism to enhance the constraint effect of the point cloud surface geometric direction information in front-end tracking and improve the stability of the alignment between the point cloud and the Gaussian map; Step 3: Establish a 3D Gaussian scene representation model that integrates visual information and point cloud information, and extract multimodal semantic features from images and point clouds to construct a unified semantic representation space; Step 4: Employ a keyframe selection method based on semantic difference awareness. Calculate the difference between the semantic features of the currently observed image and the semantic features of the rendered 3D Gaussian map to evaluate the semantic information gain of the current frame. When the semantic difference exceeds a preset threshold, the current frame is selected as a keyframe and map updates are performed; otherwise, it is only used for pose tracking. Step 5: Employ a joint optimization mechanism based on photometric consistency, geometric consistency, normal consistency, and semantic consistency constraints to construct a multimodal semantically consistent Gaussian SLAM optimization model, thereby achieving collaborative optimization of camera pose and 3D Gaussian map parameters; Step 6: Using the optimized multimodal semantic consistency Gaussian SLAM model, perform real-time 3D reconstruction and high-precision localization of the complex bridge structure environment, and output a high-completeness bridge scene map and UAV pose estimation results.

[0007] Furthermore, step one specifically includes: Step 11: Match the current frame of the LiDAR point cloud with the local reference point cloud map to establish the point cloud correspondence; Steps 1 and 2: Construct a pose optimization model based on point-to-plane geometric constraints to solve for the initial pose of the point cloud in the current frame relative to the reference map.

[0008] Furthermore, steps one and two specifically include: Let the pose transformation of the current frame point cloud relative to the local reference point cloud map be:

[0009] In the formula, For rotation matrix, The translation vector; the current frame point Transform to the reference map coordinate system to obtain the transformed points:

[0010] And define the corresponding point-to-plane residual as:

[0011] The residual represents the projection distance from the current frame point to the local plane of the reference map, and is used to measure the geometric alignment error under the current pose estimation; based on the point-to-plane residuals of all corresponding points, an optimization objective function for solving the initial pose of the current frame is established:

[0012] In the formula, The number of point pairs participating in the matching. For the first The weights of each corresponding point; since the objective function is a nonlinear optimization problem, the current pose increment is expressed in Lie algebra form:

[0013] In the formula, Indicates the translation increment. Represents the rotation increment; for the residual function Performing a first-order Taylor expansion at the current pose estimation point yields the linearized expression:

[0014] in, For the first The Jacobian matrix of each residual term with respect to the pose increment; after superimposing all residual terms, construct the normal equation:

[0015] in,

[0016] The pose increment is obtained by solving the normal equation. And update the current pose using an exponential mapping:

[0017] Repeat the point cloud matching, residual calculation, linearization solution and pose update process until the pose increment is less than the preset threshold or the maximum number of iterations is reached, and output the initial pose estimation result of the current frame relative to the local reference point cloud map.

[0018] Furthermore, step two specifically includes: Step 21: Calculate the normal vector for each sampling point in the current frame of the LiDAR point cloud. Select its local neighborhood point set Calculate the covariance matrix of the neighborhood point set:

[0019] In the formula, Find the centroid of the neighborhood point set; solve for the eigenvalues ​​and eigenvectors of the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the surface normal vector of the sampling point. , used to characterize the geometric orientation information of point clouds in local regions; Step 22: Extract the principal axis directions of the 3D Gaussian volume corresponding to the point cloud of the current frame. , and point cloud normal vector Calculate the directional consistency metric:

[0020] In the formula, the absolute value of the dot product represents the degree of similarity between the two directions. The closer it is to 1, the higher the consistency of the directions. It is used to weigh the contribution of the current point in the alignment of the point cloud and the Gaussian map. Steps two and three: Combining local Gaussian distribution density and direction consistency Constructing weighted matching weights:

[0021] The weights are used for matching and aligning the point cloud with the Gaussian map during front-end tracking, so that the surface orientation information provides stable constraints in the complex structural environment of the bridge, thereby improving the accuracy of positioning and 3D reconstruction.

[0022] Furthermore, step three specifically includes: Step 31: Inversely project each pixel of the current frame camera image and depth map into 3D space to obtain an initial 3D point cloud; generate a 3D Gaussian set based on the 3D point cloud, where each Gaussian volume contains the center position. ,radius ,color Opacity and semantic feature embedding vector To initialize a 3D Gaussian scene:

[0023] In the formula, The initial value is obtained by mapping the high-dimensional semantic features extracted from the image onto a three-dimensional Gaussian volume to ensure that the semantic features are continuously distributed in three-dimensional space. Step 32: Extract high-dimensional semantic features from the input RGB image. Simultaneously, depth information is obtained by projecting LiDAR point clouds onto the pixel plane; and by using camera intrinsic parameters... and current pose Each three-dimensional Gaussian body Render to the pixel plane to generate a rendering semantic feature map. :

[0024] In the formula, Indicates the first A Gaussian in a pixel The projection weights, The set of Gaussians projected onto this pixel; Step 33: Construct a weighted multimodal semantic representation by analyzing the rendered semantic features. semantic features of the original image Compare and calculate the semantic consistency error for each Gaussian body:

[0025] Weights are assigned to each Gaussian volume based on its density, visibility, and consistency of normal vector direction:

[0026] This constructs a weighted multimodal semantic representation space, enabling Gaussian volumes to simultaneously integrate geometric, color, and semantic information, providing a unified representation for subsequent multimodal semantic consistency SLAM rendering, keyframe selection, and joint optimization.

[0027] Furthermore, step four specifically includes: Step 41: Extract high-dimensional semantic features from the current frame camera image. And at the current estimated pose, the existing 3D Gaussian map is rendered to the corresponding viewpoint to generate a rendered semantic feature map. The semantic consistency measure is obtained by calculating the difference between observed semantic features and rendered semantic features in the feature space.

[0028] In the formula, The set of valid pixels; Step 42: Determine semantic differences Does it exceed the preset threshold? :when When the current frame is selected as a keyframe and a map update is triggered, new Gaussian volumes and semantic features are introduced; when At this time, only the current frame is used for pose tracking and not for map updates; to enhance the robustness of the decision, weights are assigned to each Gaussian volume by combining the local Gaussian distribution density and orientation consistency. This forms a weighted multimodal semantic difference measure:

[0029] Furthermore, step five specifically includes: Step 51: For the current keyframe and the reference frame, render the 3D Gaussian volume in the reference frame using camera projection to generate a pixel-level image. and compared with the actual acquired images Compare corresponding pixels and calculate the photometric consistency residual:

[0030] The sum of squared residuals of all valid pixels constitutes the photometric uniformity loss:

[0031] During the rendering process, each Gaussian volume calculates its projection weight based on its position, radius, and visibility, and these weights are accumulated to obtain the rendering value of the pixel. Step 52: Based on the front-end pose initialization results, perform nearest neighbor matching between the current frame point cloud and the corresponding points on the 3D Gaussian map, and calculate the point-to-plane residual:

[0032] In the formula, For the current frame point, For the corresponding points on the Gaussian map, As the reference point normal vector, Given the current frame pose; all residuals are weighted and summed to form the geometric consistency loss:

[0033] Step 53: Calculate the surface normal vector for each point cloud point. With respect to the principal axis direction of the corresponding Gaussian body Calculate the directional deviation:

[0034] The weighted summation of the residuals yields the normal consistency loss:

[0035] Step 54: Extract semantic features from the current frame image semantic features of the rendered 3D Gaussian map Comparison:

[0036] Weighted summation results in semantic consistency loss:

[0037] Step 55: Jointly optimize the pose and Gaussian map parameters by weighting and combining the above four types of losses to form the total loss function:

[0038] The camera pose for the current frame is iteratively optimized using a nonlinear least squares optimization method. and three-dimensional Gaussian body parameters .

[0039] Furthermore, step six specifically includes: Step 61: Optimize the parameters of the 3D Gaussian volume. It is integrated into the global Gaussian map, and the visibility and residual weights are calculated through the correspondence between point cloud and Gaussian volume. It is also weighted and fused to update based on the density and orientation consistency of local Gaussian volume to ensure map continuity and high integrity. Step 62: Use the updated 3D Gaussian map for differentiable rendering to generate the reconstructed image and semantic rendering map from the current viewpoint. At the same time, output a visual representation of the 3D geometric model to provide an intuitive basis for bridge inspection visualization, damage detection and structural analysis. Step 63: Based on the optimized camera pose The system also features a global Gaussian volume distribution, providing real-time output of the UAV's six-degree-of-freedom pose and trajectory information in the bridge environment. This information is used for subsequent path planning, keyframe selection, and incremental map construction, while ensuring high-precision positioning and continuous 3D reconstruction.

[0040] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0041] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0042] The beneficial effects of this invention are: (1) The lidar point cloud is used for front-end pose initialization to construct the initial scene geometric constraints, providing a reliable prior pose for the stable positioning of the UAV in weak texture, repetitive structure and GNSS denied environment.

[0043] (2) Perform local neighborhood analysis on the lidar point cloud, estimate the surface normal vector of the point cloud, and perform consistency characterization with the principal axis direction of the corresponding three-dimensional Gaussian body; further combine the local Gaussian distribution density and normal consistency to construct a weighted matching mechanism to improve the stability of the alignment between the point cloud and the Gaussian map and the front-end tracking accuracy.

[0044] (3) Establish a three-dimensional Gaussian scene representation model that integrates visual information and point cloud information, extract multimodal semantic features from images and point clouds, construct a unified semantic representation space, and provide multimodal support for three-dimensional reconstruction and key frame selection in complex bridge environments.

[0045] (4) A key frame selection method based on semantic difference perception is adopted. By calculating the difference between the semantic features of the current observed image and the semantic features of the 3D Gaussian map rendering, the semantic information gain of the current frame is evaluated. When the semantic difference exceeds the preset threshold, the current frame is selected as a key frame and map update is performed. Otherwise, it is only used for pose tracking, thereby effectively suppressing the computational overhead of redundant frames and ensuring the integrity of map update.

[0046] (5) A joint optimization mechanism based on photometric consistency, geometric consistency, normal consistency and semantic consistency constraints is adopted to construct a multimodal semantic consistency Gaussian SLAM optimization model to achieve collaborative optimization of camera pose and 3D Gaussian map parameters. Through residual calculation, weighted average, nonlinear least squares iterative optimization and 3D Gaussian volume parameter update, the current frame pose and Gaussian map achieve joint consistency in photometric, geometric, normal and semantic features, thereby improving positioning accuracy and reconstruction stability.

[0047] (6) Real-time 3D reconstruction and high-precision positioning are performed using the optimized multimodal semantic consistency Gaussian SLAM model, and high-completeness bridge scene map and UAV pose estimation results are output, providing intuitive basis for bridge inspection visualization, damage detection and structural analysis, while supporting continuous incremental mapping and path planning. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart of the overall process for multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0050] Figure 2 This is a framework diagram of a multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0051] Figure 3 This is a flowchart of pose initialization based on matching lidar point clouds with local maps.

[0052] Figure 4 This is a flowchart of the neighborhood analysis and normal consistency characterization of lidar point cloud.

[0053] Figure 5 It is a framework diagram of two-dimensional to three-dimensional semantic representation and three-dimensional Gaussian mapping.

[0054] Figure 6This is a keyframe selection network architecture diagram based on semantic difference awareness. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, the method of this invention first uses LiDAR point clouds for front-end pose initialization, and constructs a weighted matching mechanism by analyzing the consistency of the normal vectors of the point cloud surface and the normals of the principal axes of the 3D Gaussian volume to enhance the geometric constraints and stability of pose tracking. Secondly, it constructs a 3D Gaussian scene representation model that integrates visual and point cloud information, extracting multimodal semantic features to form a unified semantic representation space. During the mapping process, a semantic difference perception strategy based on the current observation and rendering features dynamically selects key frames for map updates. Finally, joint optimization is performed under multiple consistency constraints such as photometric, geometric, normal, and semantic aspects to achieve coordinated updates of camera pose and map parameters, thereby completing high-precision real-time positioning and high-completeness 3D reconstruction in complex bridge environments. The framework of the multimodal semantic consistency Gaussian mapping and positioning method for bridge UAV inspection is as follows: Figure 2 As shown.

[0057] Specifically, in combination Figures 1-6 This invention proposes a multimodal semantically consistent Gaussian mapping and localization method for bridge UAV inspection, the method comprising the following steps: Step 1: Initialize the front-end pose using LiDAR point cloud and construct initial scene geometric constraints to provide prior pose for stable positioning of the UAV in weak texture, repetitive structure and GNSS denied environments; Furthermore, step one specifically includes: Step 11: Match the current frame LiDAR point cloud with the local reference point cloud map to establish a point cloud correspondence; specifically, denote the current frame LiDAR point cloud as... The local reference point cloud map is denoted as For each point in the point cloud of the current frame. Search for its nearest neighbor in the local reference point cloud map. Based on the neighborhood point set of the nearest neighbor, a local plane fitting is performed to obtain the normal vector of the corresponding plane. This establishes a point-to-plane correspondence between the current frame point cloud and the local reference point cloud map.

[0058] Steps 1 and 2: Construct a pose optimization model based on point-to-plane geometric constraints to solve for the initial pose of the point cloud in the current frame relative to the reference map.

[0059] Furthermore, steps one and two specifically include: Let the pose transformation of the current frame point cloud relative to the local reference point cloud map be:

[0060] In the formula, For rotation matrix, The translation vector; the current frame point Transform to the reference map coordinate system to obtain the transformed points:

[0061] And define the corresponding point-to-plane residual as:

[0062] The residual represents the projection distance from the current frame point to the local plane of the reference map, and is used to measure the geometric alignment error under the current pose estimation; based on the point-to-plane residuals of all corresponding points, an optimization objective function for solving the initial pose of the current frame is established:

[0063] In the formula, The number of point pairs participating in the matching. For the first The weights of each corresponding point are used to suppress the influence of mismatched or unstable corresponding points on pose estimation; since the objective function is a nonlinear optimization problem, the current pose increment is expressed in Lie algebra form:

[0064] In the formula, Indicates the translation increment. Represents the rotation increment; for the residual function Performing a first-order Taylor expansion at the current pose estimation point yields the linearized expression:

[0065] in, For the first The Jacobian matrix of each residual term with respect to the pose increment; after superimposing all residual terms, construct the normal equation:

[0066] in,

[0067] The pose increment is obtained by solving the normal equation. And update the current pose using an exponential mapping:

[0068] The point cloud matching, residual calculation, linearization solution, and pose update process are repeated until the pose increment is less than a preset threshold or the maximum number of iterations is reached. The initial pose estimation result of the current frame relative to the local reference point cloud map is then output. This initial pose is used as a pose prior for subsequent 3D Gaussian map rendering, multimodal semantic alignment, and joint optimization to improve tracking stability and map construction accuracy in complex bridge structures.

[0069] Step 2: Perform local neighborhood analysis on the lidar point cloud, estimate the surface normal vector of each point, and characterize the consistency between the normal vector and the principal axis direction of the corresponding 3D Gaussian volume; further, combine the local Gaussian distribution density and normal consistency to construct a weighted matching mechanism to enhance the constraint effect of the point cloud surface geometric direction information in front-end tracking and improve the stability of the alignment between the point cloud and the Gaussian map; Furthermore, step two specifically includes: Step 21: Calculate the normal vector for each sampling point in the current frame of the LiDAR point cloud. Select its local neighborhood point set Calculate the covariance matrix of the neighborhood point set:

[0070] In the formula, Find the centroid of the neighborhood point set; solve for the eigenvalues ​​and eigenvectors of the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the surface normal vector of the sampling point. , used to characterize the geometric orientation information of point clouds in local regions; Step 22: Extract the principal axis directions of the 3D Gaussian volume corresponding to the point cloud of the current frame. , and point cloud normal vector Calculate the directional consistency metric:

[0071] In the formula, the absolute value of the dot product represents the degree of similarity between the two directions. The closer it is to 1, the higher the consistency of the directions. It is used to weigh the contribution of the current point in the alignment of the point cloud and the Gaussian map. Steps two and three: Combining local Gaussian distribution density and direction consistency Constructing weighted matching weights:

[0072] The weights are used for matching and aligning the point cloud with the Gaussian map during front-end tracking, so that the surface orientation information provides stable constraints in the complex structural environment of the bridge, thereby improving the accuracy of positioning and 3D reconstruction.

[0073] Step 3: Establish a 3D Gaussian scene representation model that integrates visual information and point cloud information, and extract multimodal semantic features from images and point clouds to construct a unified semantic representation space; Furthermore, step three specifically includes: Step 31: Inversely project each pixel of the current frame camera image and depth map into 3D space to obtain an initial 3D point cloud; generate a 3D Gaussian set based on the 3D point cloud, where each Gaussian volume contains the center position. ,radius ,color Opacity and semantic feature embedding vector To initialize a 3D Gaussian scene:

[0074] In the formula, The initial values ​​are obtained by mapping the high-dimensional semantic features extracted from the image (using the 16-dimensional vector output by DINOV2) onto a three-dimensional Gaussian volume to ensure that the semantic features are continuously distributed in three-dimensional space. Step 32: Extract high-dimensional semantic features from the input RGB image. Simultaneously, depth information is obtained by projecting LiDAR point clouds onto the pixel plane; and by using camera intrinsic parameters... and current pose Each three-dimensional Gaussian body Render to the pixel plane to generate a rendering semantic feature map. :

[0075] In the formula, Indicates the first A Gaussian in a pixel The projection weights, This is a Gaussian set projected onto the pixel; this operation maps the semantic features of the image to a three-dimensional Gaussian volume and associates the visual with the three-dimensional representation.

[0076] Step 33: Construct a weighted multimodal semantic representation by analyzing the rendered semantic features. semantic features of the original image Compare and calculate the semantic consistency error for each Gaussian body:

[0077] Weights are assigned to each Gaussian volume based on its density, visibility, and consistency of normal vector direction:

[0078] This constructs a weighted multimodal semantic representation space, enabling Gaussian volumes to simultaneously integrate geometric, color, and semantic information, providing a unified representation for subsequent multimodal semantic consistency SLAM rendering, keyframe selection, and joint optimization.

[0079] Step 4: Employ a keyframe selection method based on semantic difference awareness. Calculate the difference between the semantic features of the currently observed image and the semantic features of the rendered 3D Gaussian map to evaluate the semantic information gain of the current frame. When the semantic difference exceeds a preset threshold, the current frame is selected as a keyframe and map updates are performed; otherwise, it is only used for pose tracking. Furthermore, step four specifically includes: Step 41: Extract high-dimensional semantic features from the current frame camera image. And at the current estimated pose, the existing 3D Gaussian map is rendered to the corresponding viewpoint to generate a rendered semantic feature map. The semantic consistency measure is obtained by calculating the difference between observed semantic features and rendered semantic features in the feature space.

[0080] In the formula, The effective set of pixels is the metric that reflects the semantic information gain of the current frame relative to the existing map, indicating whether the current frame contains new structural or semantic information. Step 42: Determine semantic differences Does it exceed the preset threshold? :when When the current frame is selected as a keyframe and a map update is triggered, new Gaussian volumes and semantic features are introduced; when At this time, only the current frame is used for pose tracking and not for map updates; to enhance the robustness of the decision, weights are assigned to each Gaussian volume by combining the local Gaussian distribution density and orientation consistency. This forms a weighted multimodal semantic difference measure:

[0081] Through this mechanism, the system effectively suppresses the computational overhead of redundant frames, while ensuring the integrity and update efficiency of map construction in complex bridge structures.

[0082] Step 5: Employ a joint optimization mechanism based on photometric consistency, geometric consistency, normal consistency, and semantic consistency constraints to construct a multimodal semantically consistent Gaussian SLAM optimization model, thereby achieving collaborative optimization of camera pose and 3D Gaussian map parameters; Furthermore, step five specifically includes: Step 51: For the current keyframe and the reference frame, render the 3D Gaussian volume in the reference frame using camera projection to generate a pixel-level image. and compared with the actual acquired images Compare corresponding pixels and calculate the photometric consistency residual:

[0083] The sum of squared residuals of all valid pixels constitutes the photometric uniformity loss:

[0084] During the rendering process, each Gaussian volume calculates its projection weight based on its position, radius, and visibility, and accumulates them to obtain the rendering value of the pixel; this constraint ensures that the camera pose matches the 3D Gaussian map in color and brightness, while measuring the luminosity consistency between the current frame and the Gaussian map.

[0085] Step 52: Based on the front-end pose initialization results, perform nearest neighbor matching between the current frame point cloud and the corresponding points on the 3D Gaussian map, and calculate the point-to-plane residual:

[0086] In the formula, For the current frame point, For the corresponding points on the Gaussian map, As the reference point normal vector, Given the current frame pose; all residuals are weighted and summed to form the geometric consistency loss:

[0087] This constraint ensures the alignment of the point cloud with the Gaussian map in 3D spatial geometry. The weight of each residual... By combining local Gaussian distribution density, normal consistency, and visibility settings, the high-reliability region can contribute more to the optimization.

[0088] Step 53: Calculate the surface normal vector for each point cloud point. With respect to the principal axis direction of the corresponding Gaussian body Calculate the directional deviation:

[0089] The weighted summation of the residuals yields the normal consistency loss:

[0090] This constraint ensures that the Gaussian volume is aligned with the surface normal vector of the point cloud along the surface direction, improving the stability of local surface geometry reconstruction.

[0091] Step 54: Extract semantic features from the current frame image semantic features of the rendered 3D Gaussian map Comparison:

[0092] Weighted summation results in semantic consistency loss:

[0093] This constraint ensures the matching of camera pose and Gaussian volume in semantic feature space and reflects whether the current frame contains new structural or semantic information.

[0094] Step 55: Jointly optimize the pose and Gaussian map parameters by weighting and combining the above four types of losses to form the total loss function:

[0095] The camera pose for the current frame is iteratively optimized using a nonlinear least squares optimization method. and three-dimensional Gaussian body parameters .

[0096] In the joint optimization process, the residual values ​​for photometric consistency, geometric consistency, normal consistency, and semantic consistency are first calculated. Then, the residuals are weighted according to the local distribution density, surface normal direction consistency, and visibility of each Gaussian volume to enhance the contribution of high-reliability regions to the optimization. Subsequently, the nonlinear residual function is linearized at the current estimated pose, and a normal equation is constructed to solve for the pose increment. The camera's rotation matrix and translation vector are updated based on the solution. Simultaneously with pose update, the residuals are used to update the center position, radius, color, and semantic embedding vector of each 3D Gaussian volume using gradients, achieving coordinated adjustment of the 3D Gaussian map parameters. This iterative operation is repeated until the total loss function converges or the preset number of iterations is reached, ensuring joint consistency between the current frame camera pose and the 3D Gaussian map in terms of photometric, geometric, normal, and semantic features. Through this joint optimization process, the optimized camera pose and Gaussian volume parameters are finally output, providing a reliable multimodal fusion prior for subsequent keyframe tracking, 3D Gaussian map updates, and high-precision 3D reconstruction in complex bridge structures.

[0097] Step 6: Using the optimized multimodal semantic consistency Gaussian SLAM model, perform real-time 3D reconstruction and high-precision localization of the complex bridge structure environment, and output a high-completeness bridge scene map and UAV pose estimation results.

[0098] Furthermore, step six specifically includes: Step 61: Optimize the parameters of the 3D Gaussian volume. It is integrated into the global Gaussian map, and the visibility and residual weights are calculated through the correspondence between point cloud and Gaussian volume. It is also weighted and fused to update based on the density and orientation consistency of local Gaussian volume to ensure map continuity and high integrity. Step 62: Use the updated 3D Gaussian map for differentiable rendering to generate the reconstructed image and semantic rendering map from the current viewpoint. At the same time, output a visual representation of the 3D geometric model to provide an intuitive basis for bridge inspection visualization, damage detection and structural analysis. Step 63: Based on the optimized camera pose The system also features a global Gaussian volume distribution, providing real-time output of the UAV's six-degree-of-freedom pose and trajectory information in the bridge environment. This information is used for subsequent path planning, keyframe selection, and incremental map construction, while ensuring high-precision positioning and continuous 3D reconstruction.

[0099] For those skilled in the art, based on the ideas of this invention, the construction methods of lidar point cloud and camera image parameters, local and global geometric feature extractors, three-dimensional Gaussian volume parameters and semantic embedding vectors, keyframe semantic difference calculation, residual calculation method in joint optimization module, weight coefficient combination form, number of iterations and convergence conditions, selection of optimization algorithm and design of loss function can be appropriately changed and adjusted according to actual application scenarios.

[0100] Furthermore, the multi-scale convolutional architecture for high-resolution and low-resolution image feature streams, multi-level downsampling and sparse feature aggregation of point clouds, the weight coefficients and residual combination forms of joint photometric, geometric, normal and semantic consistency constraints in specific calculations, differentiable rendering processes and multimodal feature fusion strategies can all be flexibly adjusted according to the complexity of the bridge structure, the speed of UAV inspection and the computing resource conditions.

[0101] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0102] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection.

[0103] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0105] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0106] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0107] The multimodal semantic consistency Gaussian mapping and localization method for bridge UAV inspection proposed in this invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of this invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multimodal semantically consistent Gaussian mapping and localization method for bridge UAV inspection, characterized in that, The method includes the following steps: Step 1: Initialize the front-end pose using LiDAR point cloud and construct initial scene geometric constraints to provide prior pose for stable positioning of the UAV in weak texture, repetitive structure and GNSS denied environments; Step 2: Perform local neighborhood analysis on the lidar point cloud, estimate the surface normal vector of each point, and characterize the consistency between the normal vector and the principal axis direction of the corresponding 3D Gaussian volume; further, combine the local Gaussian distribution density and normal consistency to construct a weighted matching mechanism to enhance the constraint effect of the point cloud surface geometric direction information in front-end tracking and improve the stability of the alignment between the point cloud and the Gaussian map; Step 3: Establish a 3D Gaussian scene representation model that integrates visual information and point cloud information, and extract multimodal semantic features from images and point clouds to construct a unified semantic representation space; Step 4: Employ a keyframe selection method based on semantic difference awareness. Calculate the difference between the semantic features of the currently observed image and the semantic features of the rendered 3D Gaussian map to evaluate the semantic information gain of the current frame. When the semantic difference exceeds a preset threshold, the current frame is selected as a keyframe and map updates are performed; otherwise, it is only used for pose tracking. Step 5: Employ a joint optimization mechanism based on photometric consistency, geometric consistency, normal consistency, and semantic consistency constraints to construct a multimodal semantically consistent Gaussian SLAM optimization model, thereby achieving collaborative optimization of camera pose and 3D Gaussian map parameters; Step 6: Using the optimized multimodal semantic consistency Gaussian SLAM model, perform real-time 3D reconstruction and high-precision localization of the complex bridge structure environment, and output a high-completeness bridge scene map and UAV pose estimation results.

2. The method according to claim 1, characterized in that, Step one specifically includes: Step 11: Match the current frame of the LiDAR point cloud with the local reference point cloud map to establish the point cloud correspondence; Steps 1 and 2: Construct a pose optimization model based on point-to-plane geometric constraints to solve for the initial pose of the point cloud in the current frame relative to the reference map.

3. The method according to claim 2, characterized in that, Steps one and two specifically include: Let the pose transformation of the current frame point cloud relative to the local reference point cloud map be: In the formula, Let be a rotation matrix. The translation vector; the current frame point Transform to the reference map coordinate system to obtain the transformed points: And define the corresponding point-to-plane residual as: The residual represents the projection distance from the current frame point to the local plane of the reference map, and is used to measure the geometric alignment error under the current pose estimation; based on the point-to-plane residuals of all corresponding points, an optimization objective function for solving the initial pose of the current frame is established: In the formula, The number of point pairs participating in the matching. For the first The weights of each corresponding point; since the objective function is a nonlinear optimization problem, the current pose increment is expressed in Lie algebra form: In the formula, Indicates the translation increment. Represents the rotation increment; for the residual function Performing a first-order Taylor expansion at the current pose estimation point yields the linearized expression: in, For the first The Jacobian matrix of each residual term with respect to the pose increment; after superimposing all residual terms, construct the normal equation: in, The pose increment is obtained by solving the normal equation. And update the current pose using an exponential mapping: Repeat the point cloud matching, residual calculation, linearization solution and pose update process until the pose increment is less than the preset threshold or the maximum number of iterations is reached, and output the initial pose estimation result of the current frame relative to the local reference point cloud map.

4. The method according to claim 3, characterized in that, Step two specifically includes: Step 21: Calculate the normal vector for each sampling point in the current frame of the LiDAR point cloud. Select its local neighborhood point set Calculate the covariance matrix of the neighborhood point set: In the formula, Find the centroid of the neighborhood point set; solve for the eigenvalues ​​and eigenvectors of the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the surface normal vector of the sampling point. , used to characterize the geometric orientation information of point clouds in local regions; Step 22: Extract the principal axis directions of the 3D Gaussian volume corresponding to the point cloud of the current frame. , and point cloud normal vector Calculate the directional consistency metric: In the formula, the absolute value of the dot product represents the degree of similarity between the two directions. The closer it is to 1, the higher the consistency of the directions. It is used to weigh the contribution of the current point in the alignment of the point cloud and the Gaussian map. Steps two and three: Combining local Gaussian distribution density and direction consistency Constructing weighted matching weights: The weights are used for matching and aligning the point cloud with the Gaussian map during front-end tracking, so that the surface orientation information provides stable constraints in the complex structural environment of the bridge, thereby improving the accuracy of positioning and 3D reconstruction.

5. The method according to claim 4, characterized in that, Step three specifically includes: Step 31: Inversely project each pixel of the current frame camera image and depth map into 3D space to obtain an initial 3D point cloud; generate a 3D Gaussian set based on the 3D point cloud, where each Gaussian volume contains the center position. ,radius ,color Opacity and semantic feature embedding vector To initialize a 3D Gaussian scene: In the formula, The initial value is obtained by mapping the high-dimensional semantic features extracted from the image onto a three-dimensional Gaussian volume to ensure that the semantic features are continuously distributed in three-dimensional space. Step 32: Extract high-dimensional semantic features from the input RGB image. Simultaneously, depth information is obtained by projecting LiDAR point clouds onto the pixel plane; and by using camera intrinsic parameters... and current pose Each three-dimensional Gaussian body Render to the pixel plane to generate a rendering semantic feature map. : In the formula, Indicates the first A Gaussian in a pixel The projection weights, The set of Gaussians projected onto this pixel; Step 33: Construct a weighted multimodal semantic representation by analyzing the rendered semantic features. semantic features of the original image Compare and calculate the semantic consistency error for each Gaussian body: Weights are assigned to each Gaussian volume based on its density, visibility, and consistency of normal vector direction: This constructs a weighted multimodal semantic representation space, enabling Gaussian volumes to simultaneously integrate geometric, color, and semantic information, providing a unified representation for subsequent multimodal semantic consistency SLAM rendering, keyframe selection, and joint optimization.

6. The method according to claim 5, characterized in that, Step four specifically includes: Step 41: Extract high-dimensional semantic features from the current frame camera image. And at the current estimated pose, the existing 3D Gaussian map is rendered to the corresponding viewpoint to generate a rendered semantic feature map. The semantic consistency measure is obtained by calculating the difference between observed semantic features and rendered semantic features in the feature space. In the formula, The set of valid pixels; Step 42: Determine semantic differences Does it exceed the preset threshold? :when When the current frame is selected as a keyframe and a map update is triggered, new Gaussian volumes and semantic features are introduced; when At this time, only the current frame is used for pose tracking and not for map updates; to enhance the robustness of the decision, weights are assigned to each Gaussian volume by combining the local Gaussian distribution density and orientation consistency. This forms a weighted multimodal semantic difference measure: 。 7. The method according to claim 6, characterized in that, Step five specifically includes: Step 51: For the current keyframe and the reference frame, render the 3D Gaussian volume in the reference frame using camera projection to generate a pixel-level image. and compared with the actual acquired images Compare corresponding pixels and calculate the photometric consistency residual: The sum of squared residuals of all valid pixels constitutes the photometric uniformity loss: During the rendering process, each Gaussian volume calculates its projection weight based on its position, radius, and visibility, and these weights are accumulated to obtain the rendering value of the pixel. Step 52: Based on the front-end pose initialization results, perform nearest neighbor matching between the current frame point cloud and the corresponding points on the 3D Gaussian map, and calculate the point-to-plane residual: In the formula, For the current frame point, For the corresponding points on the Gaussian map, As the reference point normal vector, Given the current frame pose; all residuals are weighted and summed to form the geometric consistency loss: Step 53: Calculate the surface normal vector for each point cloud point. With respect to the principal axis direction of the corresponding Gaussian body Calculate the directional deviation: The weighted summation of the residuals yields the normal consistency loss: Step 54: Extract semantic features from the current frame image semantic features of the rendered 3D Gaussian map Comparison: Weighted summation results in semantic consistency loss: Step 55: Jointly optimize the pose and Gaussian map parameters by weighting and combining the above four types of losses to form the total loss function: The camera pose for the current frame is iteratively optimized using a nonlinear least squares optimization method. and three-dimensional Gaussian body parameters .

8. The method according to claim 7, characterized in that, Step six specifically includes: Step 61: Optimize the parameters of the 3D Gaussian volume. It is integrated into the global Gaussian map, and the visibility and residual weights are calculated through the correspondence between point cloud and Gaussian volume. It is also weighted and fused to update based on the density and orientation consistency of local Gaussian volume to ensure map continuity and high integrity. Step 62: Use the updated 3D Gaussian map for differentiable rendering to generate the reconstructed image and semantic rendering map from the current viewpoint. At the same time, output a visual representation of the 3D geometric model to provide an intuitive basis for bridge inspection visualization, damage detection and structural analysis. Step 63: Based on the optimized camera pose The system also features a global Gaussian volume distribution, providing real-time output of the UAV's six-degree-of-freedom pose and trajectory information in the bridge environment. This information is used for subsequent path planning, keyframe selection, and incremental map construction, while ensuring high-precision positioning and continuous 3D reconstruction.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-8.

Images