Bridge unmanned inspection system camera-lidar cycle consistency deep learning calibration method

Abstract

This invention proposes a deep learning-based calibration method for camera-LiDAR cyclic consistency in unmanned bridge inspection systems. This method creatively introduces cyclic consistency learning into the camera-LiDAR calibration task, establishing a bidirectional cyclic alignment mechanism between the image and point cloud. By optimizing the cyclic consistency constraints of cross-modal mapping, high-precision coordinate system calibration is achieved. The technical solution provided by this invention effectively solves the technical problems of traditional calibration methods, such as reliance on manual calibration objects, insufficient generalization ability, and poor dynamic adaptability. It provides an efficient and accurate calibration solution for multi-sensor systems in fields such as unmanned bridge inspection, autonomous driving, and robot navigation.

Description

Technical Field

[0001] This invention relates to the technical fields of bridge health monitoring, computer vision, visual navigation, and intelligent operation and maintenance of civil infrastructure, and in particular to a deep learning calibration method for camera-LiDAR cyclic consistency in unmanned bridge inspection systems. Background Technology

[0002] Camera-LiDAR extrinsic parameter calibration aims to establish spatial transformation relationships (such as rotation matrices and translation vectors) between the coordinate systems of the two sensors, thereby achieving accurate spatiotemporal alignment of multimodal data and providing a reliable multi-source sensing foundation for downstream tasks such as 3D target detection and autonomous navigation. Existing calibration methods can be broadly classified into three categories: target-based calibration methods, target-free calibration methods, and deep learning-based calibration methods.

[0003] (1) Target-based calibration method

[0004] Methods based on calibration objects (such as checkerboard patterns or polygonal planar boards) involve manually arranging high-precision calibration targets, extracting strongly correlated features (such as corner points and planes) from point clouds and images, and optimizing for calibration using projection consistency or distance error. While offering high accuracy, these methods have significant drawbacks:

[0005] [1] Strong scene dependence: Calibration devices (such as Apollo calibration boxes) need to be deployed in advance, and they fail in untargeted or unstructured environments, making it difficult to meet the needs of large-scale deployment of modern intelligent systems;

[0006] [2] Feature matching bottleneck: The matching of image edge features with sparse point cloud regions is easily affected by noise;

[0007] [3] Inefficient: SVD-based solution methods (such as the Umeyama algorithm) take 50-100ms per calculation, which is difficult to meet the real-time requirements of 10Hz;

[0008] [4] Initialization sensitivity: When the initial error exceeds ±10°, nonlinear optimization is prone to getting trapped in local optima.

[0009] (2) Targetless calibration method

[0010] The targetless method automatically calculates extrinsic parameters using natural environmental features or sensor motion information, avoiding reliance on manual calibration. Its process includes data acquisition, feature matching, transformation solving, and optimization verification. While applicable to dynamic scenarios such as autonomous driving, this method is still limited by the robustness of feature extraction and computational efficiency.

[0011] (3) Calibration methods based on deep learning

[0012] In contrast, novel targetless calibration methods based on deep learning directly extract high-level features from raw point cloud data using deep neural networks. Through end-to-end parameter learning, they achieve a fully automated calibration process. These methods not only eliminate reliance on manual intervention but also adapt to dynamic environmental changes, significantly improving calibration efficiency and providing a feasible technical path for the real-time deployment of intelligent sensing systems. However, existing methods generally employ direct geometric supervision strategies. This predictive approach limits the model's adaptability to calibration error ranges, particularly underperforming in scenarios with large initial biases.

[0013] In deep learning models, the cyclic consistency constraints of cross-domain data are utilized to learn bidirectional mapping relationships, providing self-supervised signals for multimodal tasks in the absence of labeled data. This approach has been successfully applied to tasks such as language-image conversion and 3D shape registration, laying the theoretical foundation for the camera-LiDAR cyclic consistency deep learning calibration method for the bridge unmanned inspection system proposed in this invention. Constructing a bidirectional mapping relationship between image and point cloud data to achieve self-supervised coordinate system calibration without manual annotation is a novel approach. However, these methods have not yet been applied to the specific technical field of camera-LiDAR calibration. Summary of the Invention

[0014] This invention proposes a deep learning-based deep learning calibration method for camera-LiDAR cyclic consistency in unmanned bridge inspection systems. This method aims to solve the challenge of online accurate calibration of six-DOF extrinsic parameters in autonomous unmanned inspection systems when external parameters of the camera-LiDAR heterogeneous sensors are missing or inaccurate. It can be widely applied in fields such as autonomous inspection of civil engineering structures (e.g., buildings, tunnels), robot synchronous localization and mapping, etc.

[0015] This invention is achieved through the following technical solution: This invention proposes a camera-LiDAR cyclic consistency deep learning calibration method for unmanned bridge inspection systems, the method comprising:

[0016] Step 1: Acquisition of camera images and LiDAR point cloud data, and generation of time-synchronized datasets;

[0017] Step 2: Establish an image depth estimation and pseudo-point cloud back-projection neural network model;

[0018] Step 3: Construct a neural network model based on a coarse-fine calibration cascade framework using a bidirectional constraint prediction mechanism based on cyclic consistency constraints;

[0019] Step 4: Employ a hierarchical training strategy to accelerate RGB branch learning and gradually optimize the overall network model;

[0020] Step 5: Use the trained cascaded predictive neural network model to calibrate the extrinsic parameters of the LiDAR and camera.

[0021] Furthermore, step one specifically includes:

[0022] Step 11: Configure a unified time reference source for the lidar and camera, establish a microsecond-level synchronization clock using the PTP protocol or GPS time signal, and mark each sensor data frame with a precise timestamp.

[0023] Steps 1 and 2: Perform data acquisition synchronization. The LiDAR acquires 3D point cloud data at a fixed frequency and records the start and end timestamps of each scanning cycle. The camera acquires image data at the same or integer multiple frequency and records the exposure midpoint timestamp of each frame.

[0024] Step 13: Establish a timestamp matching algorithm to calculate the time difference between the LiDAR scanning cycle and the camera exposure time. When the time difference is less than a preset threshold, it is determined to be a pair of synchronized data frames. For non-integer multiples of frequency, an interpolation compensation algorithm is used to generate virtual synchronized frames.

[0025] Step 14: Perform data association output, generate a unique synchronization identifier for each valid synchronization frame pair, establish a data index table to store the timestamp-data correspondence, and finally output spatiotemporally aligned multimodal data packets.

[0026] Furthermore, step two specifically includes:

[0027] Step 21: Unsupervised depth estimation of monocular sequence images based on photometric consistency, specifically including:

[0028] (1) Input: Single-frame RGB image Adjacent frame images Camera intrinsic parameter matrix ;

[0029] Camera intrinsic matrix , , The focal lengths are in the x and y directions; The coordinates of the optical center;

[0030] (2) A fully convolutional network encoder-decoder architecture is used to directly predict dense depth maps;

[0031] (3) Self-supervised signal construction;

[0032] Step 22: Backproject the depth image;

[0033] First, an image depth estimation neural network model generates initial depth prediction results. Then, based on camera imaging geometry, a rigorous back-projection transformation converts the 2D depth map into 3D pseudo-point cloud data. Specifically, the implementation first utilizes the camera's calibration parameters to perform 3D reconstruction calculations for each valid pixel and its corresponding depth value. The complete projection formula is as follows:

[0034] (9)

[0035] (10)

[0036] (11)

[0037] In the formula, This represents the three-dimensional coordinates of the target point in the camera coordinate system. These are the coordinates of the effective pixels in the image pixel coordinate system. The effective pixel depth value. For camera focal length, It is the optical center.

[0038] Furthermore, step three specifically includes:

[0039] Step 31: Using the original RGB image and 3D point cloud data as parallel inputs, the joint prediction of the initial rotation matrix and translation vector is achieved through a heterogeneous dual-branch network architecture;

[0040] Step 32: In the final prediction stage, the point cloud to depth map is converted bidirectionally through data reparameterization, and depth map constraints and point cloud constraints are introduced to jointly optimize the initial prediction results.

[0041] Furthermore, step four specifically includes:

[0042] Step 41: First layer optimization of the overall network model. Train the network as a whole, and use targeted loss functions for different modules;

[0043] Step 4.2: Accelerate RGB branch learning in the second layer.

[0044] Furthermore, in step four-one, after obtaining the rotation matrix and translation vector of the initial and final prediction transformations, the two transformations are superimposed to obtain the predicted translation vector. Calculate the smoothness between it and the inverse transform of the translational part of the random transform parameters. Loss, used to represent translation loss;

[0045] (19)

[0046] In the formula, smooth The losses are as follows:

[0047] (20)

[0048] For rotational loss, the difference in rotational quaternions is used as the rotational angle error:

[0049] (twenty one)

[0050] In the formula, For true rotational quaternions, To predict rotational quaternions;

[0051] The total predictive regression loss combines the two regression losses mentioned above and is defined as follows:

[0052] (twenty two)

[0053] In the formula, , These are the weighting coefficients.

[0054] Furthermore, step four two includes five stages, specifically:

[0055] The first stage involves applying the initial and final prediction transformation matrices of the first layer to the original point cloud for projection, thereby projecting more point cloud data onto the image plane.

[0056] The second stage: Prioritize training the RGB feature extraction branch, and optimize the visual feature representation capability separately through image data;

[0057] Phase 3: Based on the fixed initial prediction network parameters, jointly train the cross-modal fusion module and the point cloud processing branch;

[0058] Phase 4: An alternating optimization strategy is adopted, periodically unfreezing the RGB branch parameters for fine-tuning, while maintaining collaborative training of other modules;

[0059] Phase 5: Implement end-to-end joint training and simultaneously optimize all network parameters using the backpropagation algorithm.

[0060] Furthermore, step five specifically includes:

[0061] Step 51: Synchronize the acquired raw images and point cloud data in time;

[0062] Step 52: Preprocess the synchronized raw data to generate a pseudo-point cloud;

[0063] Step 53: Input the preprocessed data into the cascaded predictive neural network model and output calibration parameters;

[0064] Step 54: Evaluation and visualization of calibration results.

[0065] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the camera-lidar cyclic consistency deep learning calibration method of the bridge unmanned inspection system.

[0066] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the camera-lidar cyclic consistency deep learning calibration method of the bridge unmanned inspection system.

[0067] This invention addresses the problems of insufficient robustness in the initial calibration error range, excessive calibration iterations, and insufficient calibration accuracy in existing image-point cloud neural network prediction models. It proposes a deep learning-based deep learning calibration method for camera-LiDAR cyclic consistency in unmanned bridge inspection systems, offering the following improvements:

[0068] (1) An end-to-end automatic calibration method for lidar-camera based on a cyclic consistency network is adopted. Through deep learning, the reliance on traditional manual calibration objects is completely eliminated, and efficient online automatic calibration is achieved.

[0069] (2) By utilizing the cyclic consistency constraint, adding point cloud geometric constraints and depth map feature constraints, autonomous unsupervised prediction is achieved, which enhances scene adaptability and is less affected by noise.

[0070] (3) Optimize the network prediction process. A cascaded prediction network is adopted. Through the coarse-fine calibration process, the initial coarse prediction of the network is first realized in the feature space. Then, the coarse prediction is corrected by the cycle consistency constraint. Under a single iteration, the robustness of the model to different initial error ranges is improved.

[0071] (4) The image branch adopts a multi-resolution parallel architecture. By maintaining the parallel interaction between the high-resolution feature stream and the low-resolution semantic stream, it significantly improves the ability to capture image details and textures, while enhancing the ability to express high-dimensional features.

[0072] (5) The point cloud branch extracts local and global geometric features step by step through multi-level downsampling and sparse feature aggregation, which enhances the model’s robust perception of dynamic environments (such as moving objects and changes in viewpoint).

[0073] (6) The training method effectively solves the technical problems of slow training speed and difficulty in convergence of the visual branch in multimodal networks by adopting a hierarchical and progressive parameter optimization strategy, thereby improving training efficiency while ensuring model accuracy. In particular, the dual-layer iterative mechanism avoids the gradient conflict problem caused by direct end-to-end training and overcomes the risk of suboptimal solutions brought about by traditional staged training, providing an efficient network optimization scheme for multi-sensor fusion systems.

[0074] (7) A network architecture based on cyclic consistency constraints is adopted, new point cloud geometric constraints and depth feature constraints are introduced, and the external parameters are dynamically corrected through a multi-stage error feedback mechanism based on the initial prediction results, which significantly improves the calibration accuracy and robustness.

[0075] (8) A hierarchical and progressive prediction architecture is adopted to achieve high-precision prediction in a single iteration, and can be extended to multiple iteration scenarios to further improve the performance of the model. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0077] Figure 1 This is a flowchart of the camera-LiDAR cyclic consistency deep learning calibration method for an unmanned bridge inspection system.

[0078] Figure 2 This is a diagram of the fully convolutional encoder-decoder network architecture, which generates point clouds from RGB images to depth maps.

[0079] Figure 3 This is a diagram of a cascaded cross-modal calibration framework based on cyclic consistency constraints.

[0080] Figure 4 This is a schematic diagram of a multi-scale image feature extraction encoder for an image feature fusion network.

[0081] Figure 5 This is a schematic diagram of the downsampling module structure of the image feature fusion network encoder.

[0082] Figure 6 This is a schematic diagram of a cross-modal feature fusion and joint prediction architecture based on image-point cloud dual branches.

[0083] Figure 7 This is a schematic diagram of the coarse-fine calibration optimization process in a single iteration.

[0084] Figure 8 This is a schematic diagram of multi-scale feature modality fusion in the second-stage image feature fusion network.

[0085] Figure 9 This is a schematic diagram of a hierarchical training strategy for a network based on a cyclic consistency coarse-fine calibration cascade framework.

[0086] Figure 10 This is a visualization of the network hierarchical training strategy based on the cyclic consistency coarse-fine calibration cascade framework. Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] Combination Figures 1-10 This invention proposes a deep learning-based deep learning calibration method for camera-LiDAR cyclic consistency in an unmanned bridge inspection system, the method comprising:

[0089] Step 1: Acquisition of camera images and LiDAR point cloud data, and generation of time-synchronized datasets;

[0090] Step 2: Establish an image depth estimation and pseudo-point cloud back-projection neural network model;

[0091] Step 3: Construct a neural network model based on a coarse-fine calibration cascade framework using a bidirectional constraint prediction mechanism based on cyclic consistency constraints;

[0092] Step 4: Employ a hierarchical training strategy to accelerate RGB branch learning and gradually optimize the overall network model;

[0093] Step 5: Use the trained cascaded predictive neural network model to calibrate the extrinsic parameters of the LiDAR and camera.

[0094] Step one specifically includes:

[0095] Step 11: Configure a unified time reference source for the lidar and camera, establish a microsecond-level synchronization clock using the PTP protocol or GPS time signal, and mark each sensor data frame with a precise timestamp.

[0096] Steps 1 and 2: Perform data acquisition synchronization. The LiDAR acquires 3D point cloud data at a fixed frequency (e.g., 10Hz) and records the start and end timestamps of each scanning cycle. The camera acquires image data at the same or integer multiple frequency and records the exposure midpoint timestamp of each frame.

[0097] Step 13: Establish a timestamp matching algorithm to calculate the time difference between the LiDAR scanning cycle and the camera exposure time. When the time difference is less than a preset threshold (e.g., ±5ms), it is determined to be a pair of synchronized data frames. For non-integer multiples of frequency, an interpolation compensation algorithm is used to generate virtual synchronized frames.

[0098] The calculation process of the timestamp matching algorithm described in steps one and three is as follows.

[0099] Define the set of timestamps for LiDAR scanning cycles as follows:

[0100] (1)

[0101] In the formula, This is the start time of the lidar scan. This is the end time of the lidar scan.

[0102] Define the camera exposure timestamp set as:

[0103] (2)

[0104] In the formula, This is the midpoint exposure time for the camera.

[0105] Time alignment criteria:

[0106] (3)

[0107] In the formula, Represents the interpolation coefficients within the scan period. Take the preset time threshold .

[0108] For frames that do not meet the synchronization conditions, virtual synchronization data is generated using linear interpolation:

[0109] (4)

[0110] In the formula, and Indicates adjacent lidar scan points. and This indicates the start timestamp of two adjacent LiDAR scan frames. This indicates the generation of virtual synchronized point cloud data.

[0111] Step 14: Perform data association output, generate a unique synchronization identifier for each valid synchronization frame pair, establish a data index table to store the timestamp-data correspondence, and finally output spatiotemporally aligned multimodal data packets.

[0112] Step two specifically includes:

[0113] Step 21: Unsupervised depth estimation of monocular sequence images based on photometric consistency, specifically including:

[0114] (1) Input: Single-frame RGB image Adjacent frame images Camera intrinsic parameter matrix (Including focal length, principal point, etc.);

[0115] Camera intrinsic matrix , , The focal length (in pixels) in the x and y directions; The coordinates of the optical center;

[0116] (2) A fully convolutional network encoder-decoder architecture is used to directly predict dense depth maps;

[0117] (3) Self-supervised signal construction;

[0118] [1] Utilizing adjacent frames , Provide monitoring signals:

[0119] Through lightweight pose network Predict the relative pose of adjacent frames (SE3 transform):

[0120] (5)

[0121] In the formula, The parameters for the 6-DOF SE3 transformation are: These represent translation and rotation (expressed as axis angles), respectively. It is a lightweight pose prediction network (5-layer CNN + 2-layer fully connected).

[0122] [2] Differentiable image distortion:

[0123] (6)

[0124] In the formula, For reconstruction Frame image. For the distortion operation, the specific steps are as follows:

[0125] Step 1: Perform 3D projection

[0126] right Each pixel , calculate its in Frame coordinates :

[0127] (7)

[0128] In the formula, Representing a depth map In pixel coordinates The depth value at that location.

[0129] Step 2: Perform bilinear sampling on the depth map:

[0130] from Mid-roll value .

[0131] [3] Photometric reconstruction loss:

[0132] (8)

[0133] In the formula, The L1 norm (absolute error) is calculated separately for the three RGB channels and then the average value is taken. The minimum reconstruction error between the preceding and following frames is taken to mitigate the impact of occlusion.

[0134] Step 22: Backproject the depth image;

[0135] First, an image depth estimation neural network model generates initial depth prediction results. Then, based on camera imaging geometry principles, a rigorous back-projection transformation converts the 2D depth map into 3D pseudo-point cloud data. Specifically, this process first utilizes the camera's calibration parameters (including focal length). , and Optical Center , For each valid pixel and its corresponding depth value, a 3D reconstruction calculation is performed. The complete projection formula is:

[0136] (9)

[0137] (10)

[0138] (11)

[0139] In the formula, This represents the three-dimensional coordinates of the target point in the camera coordinate system. These are the coordinates of the effective pixels in the image pixel coordinate system. The effective pixel depth value. For camera focal length, It is the optical center.

[0140] Step three specifically includes:

[0141] Step 31: Using the original RGB image and 3D point cloud data as parallel inputs, a heterogeneous dual-branch network architecture is used to jointly predict the initial rotation matrix and translation vector; specifically including:

[0142] (1) Extract multi-scale feature representations of the input image through a parallel multi-resolution convolutional network;

[0143] A dual-branch feature extraction strategy is used in the initial prediction stage. For example... Figure 4 As shown, in the image processing branch, an image processing branch is constructed, which contains four parallel resolution feature extraction streams of 1 / 4, 1 / 8, 1 / 16 and 1 / 32. Each resolution branch maintains independent forward propagation. Only in the final output stage is the feature map size unified by bilinear interpolation and channel dimension stitching performed to ultimately achieve the preservation of high-resolution detailed features and avoid redundant feature fusion calculations.

[0144] like Figure 4 As shown, the upsampling module of the feature extraction network is implemented as follows:

[0145] [1] Two-dimensional convolutional layer (Conv2d), using kernel size, Step size and Fill in the parameters to process the input feature map of size h×w×c;

[0146] [2] Batch normalization layer (BN) normalizes the output h×w×c / n size feature map;

[0147] [3] Upsample layer (Upsample×n) increases the feature map resolution by n times to size nh×nw×c / n;

[0148] Where n is the preset upsampling factor, and the number of channels in each level of feature map is compressed at a ratio of 1 / n.

[0149] like Figure 5 As shown, this is the downsampling module of the feature extraction network, and its implementation process is as follows:

[0150] [1] The left branch includes: Max Pooling, 1×1 Convolutional layer, and Batch Normalization layer.

[0151] [2] The intermediate branch contains: a 3×3 convolutional layer (Conv3×3) with stride s=2, and a batch normalization layer (Batch Norm).

[0152] [3] The right branch includes: Max Pooling, 1×1 Convolutional layer, and Sigmoid activation function;

[0153] [4] The feature maps of the left and middle branches are superimposed and then multiplied with the feature map of the right side, and finally the LeakyReLU activation function is applied.

[0154] Hierarchical compression of feature maps is achieved through branching processes, specifically as follows:

[0155] (1) Geometric feature representation of the input point cloud is extracted using a PointNet++ hierarchical structure;

[0156] In the point cloud processing branch, an improved PointNet++ hierarchical structure is adopted to progressively extract 128-dimensional global geometric features through farthest point sampling and local feature aggregation. The specific process is as follows:

[0157] [1] Farthest point sampling (FPS):

[0158] (12)

[0159] In the formula, For the first Point cloud after layer sampling For target points, FPS ( The iterative selection selects the point furthest from the already selected points to ensure spatial uniformity.

[0160] [2] Local feature aggregation:

[0161] Step 1: Neighborhood Definition (KNN):

[0162] (13)

[0163] In the formula, For point Given the 20 nearest neighbors, topk(·, k) returns the top 20 neighbors with the smallest distance. k One point.

[0164] Step 2: Feature aggregation:

[0165] (14)

[0166] In the formula, For the first Layers The feature vector, [·] is the vector concatenation operation (coordinate difference + feature). It is a learnable multilayer perceptron with parameters of MAXPOOL is an operation that takes the maximum value along the neighborhood dimension.

[0167] [3] Hierarchical jump connections:

[0168] The internet has L Layer, number l Layered cloud P (l) The corresponding features are The jump connection operation is as follows:

[0169] (15)

[0170] In the formula, For the features calculated by the current layer through local aggregation, Features of the next higher layer (coarser resolution), For feature interpolation operations, These are the output features after fusion.

[0171] (2) Image and point cloud feature aggregation

[0172] The features output from the two branches are reduced in dimensionality by 1×1 convolution and then fed into the cross-modal multi-head attention module for deep interaction. Finally, the predicted rotation quaternions and translation vectors are output through the decoupled regression heads.

[0173] Figure 6 The process of fusing point cloud features with image features was visualized: accurate calibration is built upon a strong feature learning capability, namely, obtaining fused features from the image multi-scale feature aggregation module, as well as point cloud features, which are then input into DenseNet for further feature learning. Subsequently, a branch of one fully connected (FC) layer and two stacked FC layers is used to predict and obtain a 1×3 translation vector and a 1×4 rotation quaternion.

[0174] Step 32: In the final prediction stage, bidirectional conversion between point cloud and depth map is achieved through data reparameterization, and depth map constraints and point cloud constraints are introduced to jointly optimize the initial prediction results. The specific process is as follows:

[0175] (1) Depth map reconstruction and monocular pseudo-point cloud generation:

[0176] The original point cloud is transformed from the world coordinate system to the camera coordinate system using the rotation matrix and translation vector obtained from the initial prediction, and a depth map is generated by perspective projection using the camera intrinsic matrix (this process is the inverse of step 22). In the point cloud branch, an image depth estimation neural network model is used to estimate depth information from the RGB image, and a pseudo point cloud is generated by combining it with the camera intrinsics.

[0177] (2) Feature extraction from depth map and pseudo-point cloud:

[0178] The feature extraction stage uses an encoder that is isomorphic to the initial stage but has independent parameters: the depth map encoder has the same structure as the RGB encoder but does not share parameters, and the pseudo point cloud encoder has the same architecture as the original point cloud encoder but with independent parameters.

[0179] (3) Hierarchical cross-modal feature fusion:

[0180] Feature fusion employs a hierarchical design, first performing intra-modal feature enhancement (fusion of depth map features with initial image features, and fusion of pseudo-point cloud features with initial point cloud features), followed by a final cross-modal fusion. The final parameter prediction network maintains the same structure as the initial stage, outputting refined 6-DoF calibration parameters.

[0181] Figure 8 This demonstrates the process of intramodal feature fusion across multiple scales of image features in an image branch. The specific process is as follows:

[0182] [1] Input feature map:

[0183] The module receives multi-resolution feature maps from the feature extraction network, including four feature maps at different scales (e.g., 250×128×32, 128×64×64, 32×16×256, 16×8×512). These feature maps correspond to representations from high resolution to low resolution, respectively.

[0184] [2] Feature splicing (CONCAT):

[0185] At each resolution level, feature maps from the RGB image branch and the depth map branch are concatenated along the channel dimension (CONCAT) to fuse information from both modalities.

[0186] [3] Downsampling and fusion:

[0187] High-resolution feature maps are progressively downsampled using 3×3 convolutions with a stride of 2, aligning them with low-resolution feature maps. Feature maps of all resolutions are then initially fused through element-wise summation.

[0188] [4] Attention weight generation:

[0189] Global pooling is performed on the fused feature maps to generate a global descriptive vector. Attention weights for each resolution branch are calculated using a multilayer perceptron (MLP, which includes fully connected layers and softmax) to dynamically adjust the importance of features at different resolutions.

[0190] [5] Feature recalibration:

[0191] Multiplying the original multi-resolution feature map with its corresponding attention weights (multiplication) achieves feature recalibration, highlighting important information.

[0192] [6] Cross-feature level fusion:

[0193] The cross-feature-level fusion module is used to further integrate features of different resolutions, and finally outputs a unified multi-scale feature representation.

[0194] The fusion process between pseudo-point cloud features and initial point cloud features is relatively simple, and intra-modal fusion is performed by stitching together the features.

[0195] Finally, the features fused within the modality are input into a network similar to the initial fusion stage, and the refined 6-DoF calibration parameters are output.

[0196] Neural network architectures based on coarse-fine calibration cascade frameworks, such as... Figure 3 As shown, the network adopts a cascaded design, which includes two key stages: initial prediction and final prediction. Each stage consists of feature extraction and feature fusion modules.

[0197] The network processing flow is as follows: The network takes the original RGB image and LiDAR point cloud as input data. First, it undergoes data preprocessing, initial prediction, and point cloud projection operations to generate two derived data modalities—image pseudo-point cloud and depth map. Subsequently, these four data modalities (original image, original point cloud, pseudo-point cloud, and depth map) are input into the network for the second-stage feature fusion, and finally output the 6-DOF extrinsic parameter calibration results.

[0198] Step four specifically includes:

[0199] Step 41: First layer optimization of the overall network model. The entire network is trained, with different loss functions applied to different modules; specifically:

[0200] (1) The initial prediction network measures the proximity between the two types of features by calculating the cosine similarity between the features extracted from the initial point cloud and the features of the RGB image:

[0201] (16)

[0202] In the formula, It is an image feature vector. It is a point cloud feature vector. In the method described in this invention, the cosine similarity loss is applied four times, including the cosine similarity loss between the image feature vector and the point cloud feature vector of the initial prediction network, the cosine similarity loss between the fused image feature vector and the fused point cloud feature vector, the cosine similarity between the image pseudo-point cloud features and the transformed original point cloud features, and the cosine similarity between the depth map features and the image features.

[0203] In addition, to prevent overfitting, a method was introduced during training. Regularization term.

[0204] (2) Depth Map Branch: This branch introduces additional supervision information using the concept of cycle consistency. The original point cloud is transformed twice, through initial prediction transformation and final prediction transformation, to the camera coordinate system. A depth map is generated using the camera intrinsic parameters projection. The accuracy of the cascaded prediction result is measured by calculating the dense pixel photometric error between the depth map features and the target depth map features.

[0205] (17)

[0206] In the formula, To predict the pixel values ​​of the depth map, For the true depth map, This represents the number of pixels.

[0207] (3) Point cloud branch: This branch also uses the idea of ​​cycle consistency to introduce additional supervision information. The original point cloud is transformed by the initial prediction and the final prediction, and the chamfer distance (CD) loss of the target point cloud is used to measure the accuracy of the initial prediction of the cascaded prediction result.

[0208] (18)

[0209] In the formula, To predict point clouds, For realistic point clouds, To predict the number of point clouds, This represents the actual number of point clouds.

[0210] (4) Final prediction network: In this network, a prediction regression loss is introduced, and the fused depth map branch features and the fused point cloud branch features are fused again across modal features, and finally the final prediction transformation parameters are output.

[0211] After obtaining the rotation matrices and translation vectors of the initial and final prediction transformations, the two transformations are superimposed to obtain the predicted translation vector. Calculate the smoothness between it and the inverse transform of the translational part of the random transform parameters. Loss, used to represent translation loss;

[0212] (19)

[0213] In the formula, smooth The losses are as follows:

[0214] (20)

[0215] For rotational loss, the difference in rotational quaternions is used as the rotational angle error:

[0216] (twenty one)

[0217] In the formula, For true rotational quaternions, To predict rotational quaternions;

[0218] The total predictive regression loss combines the two regression losses mentioned above and is defined as follows:

[0219] (twenty two)

[0220] In the formula, , These are the weighting coefficients.

[0221] Step 4.2: Accelerate RGB branch learning in the second layer.

[0222] Step 42 includes five stages, specifically:

[0223] The first stage involves applying the initial and final prediction transformation matrices of the first layer to the original point cloud for projection, thereby projecting more point cloud data onto the image plane.

[0224] The second stage: Prioritize training the RGB feature extraction branch, and optimize the visual feature representation capability separately through image data;

[0225] Phase 3: Based on the fixed initial prediction network parameters, jointly train the cross-modal fusion module and the point cloud processing branch;

[0226] Phase 4: An alternating optimization strategy is adopted, periodically unfreezing the RGB branch parameters for fine-tuning, while maintaining collaborative training of other modules;

[0227] Phase 5: Implement end-to-end joint training and simultaneously optimize all network parameters using the backpropagation algorithm.

[0228] Step five specifically includes:

[0229] Step 51: Synchronize the acquired raw images and point cloud data in time;

[0230] Step 52: Preprocess the synchronized raw data to generate a pseudo-point cloud;

[0231] Step 53: Input the preprocessed data into the cascaded predictive neural network model and output calibration parameters;

[0232] Step 54: Evaluation and visualization of calibration results.

[0233] As shown in Table 1, the trained cascaded predictive neural network model was used to calibrate the extrinsic parameters of the LiDAR and camera under four different initial calibration error ranges. The large error range (azimuth ±64° / displacement ±2.4m) simulated the initial uncalibrated state of the sensor system, while the small error range (azimuth ±8° / displacement ±0.3m) corresponded to the time-varying drift error of the calibrated system. The network maintained excellent performance under all test conditions, especially achieving a minimum average error of 0.030 meters in translation component estimation, demonstrating strong robustness.

[0234] Table 1. Model prediction results within four different initial calibration error ranges.

[0235]

[0236] Figure 10 To achieve this, a hierarchical training strategy was employed, training the calibration results through different rounds. In the figure, green point clouds represent the ground truth values, red point clouds represent the prediction results of the first layer, and blue point clouds correspond to the prediction results of the second layer. As shown, as training progresses, the predicted point clouds gradually converge towards the ground truth values, ultimately achieving a high degree of agreement between the prediction results and the target values.

[0237] This invention addresses the shortcomings of existing technologies by creatively introducing cyclic consistency learning into camera-LiDAR calibration tasks, establishing a bidirectional cyclic alignment mechanism between images and point clouds, and achieving high-precision coordinate system calibration by optimizing the cyclic consistency constraints of cross-modal mapping. The innovations of this method are: [1] using cyclic consistency to provide self-supervised signals, avoiding dependence on manually labeled data; [2] simultaneously optimizing the transformation process from image to point cloud and from point cloud to image through bidirectional mapping relationships, improving calibration accuracy; [3] combining global semantic similarity with local geometric feature alignment to enhance the robustness of cross-modal data association. The technical solution provided by this invention effectively solves the technical problems of traditional calibration methods such as reliance on manual calibration objects, insufficient generalization ability, and poor dynamic adaptability, providing an efficient and accurate calibration solution for multi-sensor systems in fields such as unmanned bridge inspection, autonomous driving, and robot navigation.

[0238] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the camera-lidar cyclic consistency deep learning calibration method of the bridge unmanned inspection system.

[0239] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the camera-lidar cyclic consistency deep learning calibration method of the bridge unmanned inspection system.

[0240] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0241] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0242] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0243] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0244] The above provides a detailed description of the camera-lidar cyclic consistency deep learning calibration method for the bridge unmanned inspection system proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A camera-lidar cyclic consistency deep learning calibration method for bridge unmanned inspection system, characterized in that, The method includes: Step 1: Acquisition of camera images and LiDAR point cloud data, and generation of time-synchronized datasets; Step 2: Establish an image depth estimation and pseudo-point cloud back-projection neural network model; Step two specifically includes: Step 21: Unsupervised depth estimation of monocular sequence images based on photometric consistency, specifically including: (1) input: single frame RGB image , adjacent frame image , camera intrinsic matrix ; Camera intrinsic matrix , , is the focal length in x, y direction; is the optical center coordinate; (2) A fully convolutional network encoder-decoder architecture is used to directly predict dense depth maps; (3) Self-supervised signal construction; Step 22: Backproject the depth image; First, an image depth estimation neural network model generates initial depth prediction results. Then, based on camera imaging geometry, a rigorous back-projection transformation converts the 2D depth map into 3D pseudo-point cloud data. Specifically, the implementation first utilizes the camera's calibration parameters to perform 3D reconstruction calculations for each valid pixel and its corresponding depth value. The complete projection formula is as follows: (9) (10) (11) In the formula, This represents the three-dimensional coordinates of the target point in the camera coordinate system. These are the coordinates of the effective pixels in the image pixel coordinate system. The effective pixel depth value. For camera focal length, For optical center; Step 3: Construct a neural network model based on a coarse-fine calibration cascade framework using a bidirectional constraint prediction mechanism based on cyclic consistency constraints; Step 4: Employ a hierarchical training strategy to accelerate RGB branch learning and gradually optimize the overall network model; Step 5: Use the trained cascaded predictive neural network model to calibrate the extrinsic parameters of the LiDAR and camera.

2. The method according to claim 1, characterized in that, Step one specifically includes: Step 11: Configure a unified time reference source for the lidar and camera, establish a microsecond-level synchronization clock using the PTP protocol or GPS time signal, and mark each sensor data frame with a precise timestamp. Steps 1 and 2: Perform data acquisition synchronization. The LiDAR acquires 3D point cloud data at a fixed frequency and records the start and end timestamps of each scanning cycle. The camera acquires image data at the same or integer multiple frequency and records the exposure midpoint timestamp of each frame. Step 13: Establish a timestamp matching algorithm to calculate the time difference between the LiDAR scanning cycle and the camera exposure time. When the time difference is less than a preset threshold, it is determined to be a pair of synchronized data frames. For non-integer multiples of frequency, an interpolation compensation algorithm is used to generate virtual synchronized frames. Step 14: Perform data association output, generate a unique synchronization identifier for each valid synchronization frame pair, establish a data index table to store the timestamp-data correspondence, and finally output spatiotemporally aligned multimodal data packets.

3. The method of claim 2, wherein, Step three specifically includes: Step 31: Using the original RGB image and 3D point cloud data as parallel inputs, the joint prediction of the initial rotation matrix and translation vector is achieved through a heterogeneous dual-branch network architecture; Step 32: In the final prediction stage, the point cloud to depth map is converted bidirectionally through data reparameterization, and depth map constraints and point cloud constraints are introduced to jointly optimize the initial prediction results.

4. The method according to claim 3, characterized in that, Step four specifically includes: Step 41: First layer optimization of the overall network model. Train the network as a whole, and use targeted loss functions for different modules; Step 4.2: Accelerate RGB branch learning in the second layer.

5. The method of claim 4, wherein, In step four, after obtaining the rotation matrix and translation vector of the initial and final prediction transformations, the two transformations are superimposed to obtain the predicted translation vector. Calculate the smoothness between it and the inverse transform of the translational part of the random transform parameters. Loss, used to represent translation loss; (19) wherein the smoothing Losses are as follows: (20) For rotational loss, the difference in rotational quaternions is used as the rotational angle error: (21) wherein is a true rotation quaternion, is a predicted rotation quaternion; The total predictive regression loss combines the two regression losses mentioned above and is defined as follows: (22) In the formula, , are weight coefficients.

6. The method of claim 5, wherein, Step 42 includes five stages, specifically: The first stage involves applying the initial and final prediction transformation matrices of the first layer to the original point cloud for projection, thereby projecting more point cloud data onto the image plane. The second stage: Prioritize training the RGB feature extraction branch, and optimize the visual feature representation capability separately through image data; Phase 3: Based on the fixed initial prediction network parameters, jointly train the cross-modal fusion module and the point cloud processing branch; Phase 4: An alternating optimization strategy is adopted, periodically unfreezing the RGB branch parameters for fine-tuning, while maintaining collaborative training of other modules; Phase 5: Implement end-to-end joint training and simultaneously optimize all network parameters using the backpropagation algorithm.

7. The method of claim 6, wherein, Step five specifically includes: Step 51: Synchronize the acquired raw images and point cloud data in time; Step 52: Preprocess the synchronized raw data to generate a pseudo-point cloud; Step 53: Input the preprocessed data into the cascaded predictive neural network model and output calibration parameters; Step 54: Evaluation and visualization of calibration results.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

9. A computer readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-7.

Images