A train multi-sensor automatic calibration method, an electronic device and a storage medium

Abstract

The application discloses a kind of train multi-sensor automatic calibration method, electronic equipment and storage medium, comprising the following steps: step S1: save camera and laser radar data;Step S2: the data saved in step S1 is preprocessed, and the characteristic trajectory of camera and the curved surface equation of laser point cloud map are obtained;Step S3: the image trajectory and point cloud map obtained in step S2 are associated by loss function, and the associated function of image trajectory to point cloud map is established;Step S4: according to the associated function obtained in step S3, a single camera and laser radar are optimized;Step S5: a plurality of cameras and laser radars are optimized together using the method of step S4, and the optimized calibration parameters of a plurality of cameras and laser radars are obtained;Step S6: the original calibration parameters are replaced using the optimized calibration parameters obtained in step S5.The application can realize automatic calibration, and does not need specific calibration tool.

Description

Technical Field

[0001] This invention belongs to the field of rail transit technology, specifically relating to the field of train obstacle detection technology. Background Technology

[0002] In train obstacle detection, the fusion detection of obstacles requires precise calibration of multiple sensors such as cameras and lasers. Furthermore, during train operation, some bumps can cause sensor displacement, requiring periodic sensor detection. Sensors that have shifted need to be recalibrated.

[0003] Existing calibration methods require the creation of specific calibration tools and manual operation by personnel with relevant knowledge and experience, making secondary calibration very difficult. Summary of the Invention

[0004] In view of the problems existing in the calibration of train sensors in the prior art, the technical problem to be solved by the present invention is to provide an automatic calibration method, electronic device and storage medium for multiple train sensors, which can realize automatic calibration without the need for specific calibration tools.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] On the one hand, an automatic calibration method for multiple sensors in a train is provided, including the following steps:

[0007] Step S1: Before calibration begins, save the image data acquired by the camera and the laser point cloud data acquired by the LiDAR at a set time.

[0008] Step S2: Preprocess the camera image data and lidar point cloud data saved in step S1 to obtain the camera feature trajectory and the surface equation of the lidar point cloud map.

[0009] Step S3: Establish a correlation function between the image trajectory and the point cloud map obtained in step S2 by associating them through a loss function;

[0010] Step S4: Optimize the individual camera and LiDAR based on the correlation function between the image trajectory and the point cloud map obtained in Step S3;

[0011] Step S5: Optimize multiple cameras and LiDAR together using the method in step S4 to obtain optimized calibration parameters for multiple cameras and LiDAR;

[0012] Step S6: Replace the original calibration parameters with the optimized calibration parameters obtained in step S5.

[0013] Preferably, in step S2, feature trajectories are first extracted for each camera using the SceneRF method; then, the laser point cloud data for a set time is accumulated and superimposed using the ICP algorithm, and the superimposed laser point cloud data is initialized by the self-motion estimation of the IMU, moving object point clouds are removed, ground point clouds are removed from the accumulated point cloud data using the ground removal method, and Poisson surface reconstruction is performed on the non-ground point clouds to obtain the point cloud surface.

[0014] Preferably, feature trajectory extraction for each camera using the SceneRF method includes the following steps:

[0015] Feature detection was performed using the SIFT algorithm.

[0016] Feature matching, matching and track building, pairwise matching of image pairs using Euclidean distance, a successful match is considered when the distance is less than a set threshold, false matches are removed, after all matching tracks are found, an image connectivity graph is constructed, which contains the nodes of each image and the edges of images with common tracks;

[0017] Motion structure recovery yields estimated camera parameters and scene geometry information, i.e., sparse 3D point clouds.

[0018] Preferably, step S3 includes the following steps:

[0019] Let Π i A function representing the image pixels projected onto the i-th camera from a 3D point centered at the camera;

[0020] make This represents the pose of the i-th camera, which is transformed from a camera-centered coordinate system to a laser-centered coordinate system L.

[0021] set up This indicates that the pose of L at timestamp t is transformed into the map coordinate system W.

[0022] Method lines and positions (n w ,p w Parametrically define a specific plane in the map coordinate system;

[0023] Γ operation through The image coordinates are converted to map coordinates, and the intersection point of x and the plane (n,p) is found. Geometric loss formula 2 is then defined.

[0024]

[0025] The feature normals of the plane are derived from the point cloud plane where the projected image rays intersect. Based on the initial estimates of the camera-to-LiDAR pose and camera characteristics, a loss function is used to establish the association function between the image trajectory and the point cloud map, i.e., Equation 3:

[0026]

[0027] The preferred method for optimizing a single camera and LiDAR is as follows:

[0028] First, an external optimization loop from single camera to LiDAR is used to improve the initial estimate, reject anomalous trajectories, and estimate the feature point locations for corresponding multi-camera detection.

[0029] In each iteration, step S3 is executed to remove trajectories with low correlation and use RANSAC to find the corresponding extrinsic parameters of the trajectories, making the correlation threshold between the trajectories and the map tighter. This process is repeated to add new trajectories to the initial trajectory set until the extrinsic parameters converge.

[0030] Preferably, the method for optimizing multiple cameras and LiDAR together is as follows: simultaneously project the estimated values ​​of two cameras onto the LiDAR map; then use the parameters obtained from step S4 to repeat the optimization steps of step S4 to obtain the final calibration parameters of the two cameras to the LiDAR.

[0031] Preferably, the time in step S1 is set to 10 seconds.

[0032] Preferably, an automatic calibration time interval is set, and the above-mentioned automatic calibration method for multiple train sensors is automatically executed at the set time.

[0033] In another aspect, an electronic device is provided, the electronic device comprising:

[0034] The memory stores multiple instructions;

[0035] A processor that loads instructions from the memory to execute steps in the automatic calibration method.

[0036] A computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by an electronic device, implements the steps in the automatic calibration method described above.

[0037] The technical solution adopted in this invention has the following beneficial effects:

[0038] 1. Joint calibration is performed through explicit camera cross-constraints to ensure accurate extrinsic parameters between cameras and between cameras and lasers, thus eliminating the need for camera intrinsic parameter calibration.

[0039] 2. Calibration is performed based on image data collected by the camera and laser point cloud data collected by the lidar. No specific calibration tools are required; calibration is performed using the external environment.

[0040] 3. The automatic calibration time interval is set, and the automatic calibration method of train multi-sensor is automatically executed at the set time, thus realizing automatic calibration at regular intervals without manual intervention.

[0041] The specific technical solutions adopted in this invention and their beneficial effects will be disclosed in detail in the following specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 This is a flowchart of the automatic calibration method for multiple sensors on trains;

[0044] Figure 2 This is a schematic diagram of projection error. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0046] Example 1

[0047] This embodiment relates to an automatic calibration method for multiple sensors on a train. It employs explicit camera cross-constraints for joint calibration to ensure accurate extrinsic parameters between cameras and between cameras and lasers. Figure 1 As shown, it includes the following steps:

[0048] Step S1: Trigger calibration requirements and save the image data acquired by the camera and the laser point cloud data acquired by the lidar for a set period of time before calibration begins;

[0049] Step S2: Preprocess the camera image data and lidar point cloud data saved in step S1 to obtain the camera feature trajectory and the surface equation of the lidar point cloud map.

[0050] Step S3: Establish a correlation function between the image trajectory and the point cloud map obtained in step S2 by associating them through a loss function;

[0051] Step S4: Optimize the individual camera and LiDAR based on the correlation function between the image trajectory and the point cloud map obtained in Step S3;

[0052] Step S5: Optimize multiple cameras and LiDAR together using the method in step S4 to obtain optimized calibration parameters for multiple cameras and LiDAR;

[0053] Step S6: Replace the original calibration parameters with the optimized calibration parameters obtained in step S5.

[0054] The key steps involved in this embodiment are explained in detail below:

[0055] In step S1, after receiving the calibration request, the images and laser point cloud data of each camera are saved for 10 seconds. Of course, this time can also be adjusted.

[0056] Preprocessing of camera images and laser point cloud data

[0057] 1. Feature trajectories are extracted for each camera using the SceneRF method.

[0058] SceneRF is an offline algorithm for 3D reconstruction based on various collected unordered images. It first extracts focal length information from the images (necessary for initializing Bundle Adjustment), then uses feature extraction algorithms such as SIFT to extract image features. A kd-tree model is used to calculate the Euclidean distance between feature points in two images for feature point matching, thus finding image pairs with a required number of matching feature points. For each image pair, epipolar geometry is calculated, the F matrix is ​​estimated, and the Ransac algorithm is used to optimize and improve the pair. If feature points can be chained through these pairs and continuously detected, a trajectory can be formed.

[0059] Next, we move to the structure-from-motion (SFM) section. The crucial first step is selecting a suitable image pair to initialize the entire image base (BA) process. First, the initial BA is performed on the two selected images. Then, new images are added in a loop for further BA, until no more suitable images can be added, at which point the BA process ends. This yields estimated camera parameters and scene geometry information, i.e., a sparse 3D point cloud. The bundle adjustment between the two images uses the Sparse Bundle Adjustment (SBA) software package, a nonlinear least squares optimization algorithm. The specific algorithm consists of the following three steps:

[0060] 1.1 Feature Detection

[0061] For the feature detection step, the SIFT descriptor, which is scale and rotation invariant, is used. It is robust and suitable for extracting image feature point information under various scale transformations and rotation angles. It is also highly accurate and has advantages in offline algorithms where time cost is not a concern.

[0062] The SIFT algorithm calculates the position information (x, y) of feature points using Gaussian filters (DOG) of different sizes. It also provides a descriptor: a 4x4 histogram around a feature point, where each histogram contains gradient directions for 8 bins, resulting in a 4x4x8 = 128-dimensional feature vector. However, the scale and orientation information calculated by the SIFT algorithm are not used.

[0063] 1.2 Feature Matching

[0064] The second step is matching and establishing tracks. Image pairs are matched pairwise, generally using Euclidean distance, and there are two methods:

[0065] Brute-force matching exhaustively calculates the distance to all feature points;

[0066] Nearest neighbor search, by building a KD tree, can narrow down the search range and improve efficiency, but it may not be optimal. Therefore, the value of the neighborhood is crucial; the larger the value, the more accurate it is, but the larger the value, the greater the computational cost.

[0067] Once the feature points of each image are extracted, pairwise feature point matching between images is required. Let F(I) represent the feature points around image I. For each image pair I and J, consider each feature f∈F(I) and find the nearest neighbor feature vector fnn∈F(J):

[0068]

[0069] In fact, the algorithm uses a kd-tree data structure to calculate nearest neighbor matching. Then, let the distance between the nearest neighbors be d1, and the distance between the second nearest matching points be d2. If the ratio of the two distances d1 and d2 is less than a threshold, such as 0.6, it can be considered an acceptable matching pair. Thus, a feature point in image I will have at most one matching feature point in image J, but image J may match multiple feature points in image I, resulting in a many-to-one situation. In reality, feature points should correspond one-to-one. Therefore, an algorithm is needed to remove duplicate feature point matching pairs to handle this many-to-one situation. Finally, if the number of feature point matches between two images is no less than 16, it is considered a preliminary image pair.

[0070] A match is considered successful when the distance is less than a certain threshold, but there are also many false matches, requiring multiple methods to remove them:

[0071] If the ratio of the nearest distance to the second nearest distance is greater than a certain threshold, it should be removed.

[0072] The fundamental matrix is ​​calculated using the RANSC eight-point sampling consensus algorithm for the matching points, and matching pairs that do not meet the fundamental matrix are eliminated.

[0073] However, the initial matching pairs may still be unreliable, requiring further testing using geometric constraints. This test is based on a real-world scenario assuming a static scene; not all matching feature points conform to physical laws in the actual scene. Therefore, epipolar geometry needs to be calculated. The F matrix can link the pixel coordinates between two images and contains camera intrinsic information. The pixel coordinates of each matching pair must satisfy the following:

[0074]

[0075] The F matrix calculation results in a lot of noisy data, so it needs to be filtered using the RANSAC (Random Sample Consensus) algorithm. The RANSACA hypothesis is made using the 8-point method, where the threshold for the number of outliers should be less than 0.6% of the image's length and width.

[0076] Once all pairwise matching image pairs are determined, we can connect the common feature matching points that appear in multiple images to form a trajectory. For example, feature f1∈F(I1) matches feature f2∈F(I2), and f2 matches feature f3∈F(I3). These features can form a trajectory {f1, f2, f3}. Then, we use breadth-first search (BFS) to find the complete trajectory of each feature point in all image pairs.

[0077] Once all matching trajectories are found, an image connectivity graph is constructed, containing nodes for each image and edges of images sharing a common trajectory.

[0078] 1.3 Structure from motion

[0079] The extrinsic parameters of the camera are described using a 3x3 rotation matrix R and a 1x3 translation vector (or the camera center coordinate vector). The intrinsic parameters of the camera are described using a focal length f and two radial distortion parameters k1 and k2. The geometric scene provides each 3D point Xj in the trajectory. Through the projection equation, a 3D point Xj is projected onto the camera's 2D image plane. The projection error is the distance between the projected point and the ground truth point in the image, such as... Figure 2 As shown.

[0080] For n viewpoints and m trajectories, the objective optimization equation for projection error can be written as:

[0081]

[0082] When camera i observes trajectory j, Wij is set to 1; otherwise, it is set to 0. ||qij-P(Ci,Xj)|| is the cumulative sum of projection errors of trajectory j in camera i. The goal of the SceneRF algorithm is to find suitable camera and scene parameters to optimize this objective function. g is solved using a nonlinear least squares optimization method, the well-known one being bundle adjustment.

[0083] Choosing the right initial image pair is crucial. Incorrect initialization can lead to local optima, causing the subsequent Base Algorithm (BA) to enter an infinite loop and preventing the correct solution from reaching the global optimum. Specifically, there are two requirements: first, a sufficient number of matching points; second, a sufficiently distant camera center.

[0084] Specifically, a homography model between two image transformations is used here to find initial image pairs. If the homography model does not fit well, it indicates that there is still a certain distance between the camera center and the image. The RANSAC method is also used for noise reduction to improve the reliability of the matching, aiming to select a low percentage of inliers, but ensuring at least 100 matching inliers.

[0085] The system uses a 5-point method to estimate the extrinsic parameters of the initial matching pairs. After trajectory triangulation, it can provide initial 3D points. The first bundle adjustment can then begin with the initial two frames of images. Sparse bundle adjustment (SBA) is used here. Bundle adjustment is an iterative process. After each iteration, all 3D points are back-projected to the pixel coordinates of their respective images and compared with the initial coordinates. If the coordinates are greater than a certain threshold, they should be removed from the track. If there are fewer than two such points in the track, the entire track is removed. This process continues until no points can be removed.

[0086] Finally, new cameras and 3D points are continuously added for basis analysis (BA). This process continues until the remaining cameras have observed no more than 20 points, indicating that there are not enough points left to add, at which point the BA ends. The resulting camera estimation parameters and scene geometry information are then obtained, i.e., a sparse 3D point cloud.

[0087] 2. Create a point cloud map

[0088] The ICP algorithm is used to accumulate and overlay the lidar data for 10 seconds. The point cloud data of the overlaid lidar is initialized by the self-motion estimation of the IMU to remove the point cloud of moving objects. The accumulated point cloud data is then removed by the ground removal method, and the non-ground point cloud is reconstructed by Poisson surface to obtain the point cloud surface representation (Equation 1).

[0089]

[0090] Associated image feature trajectories and point cloud maps

[0091] Let Π i This represents the function that projects a 3D point centered at the camera onto the image pixels of the i-th camera; it is simply a function of the camera's intrinsic parameters. Let... This represents the pose of the i-th camera, transformed from a camera-centric coordinate system to a laser-centric coordinate system L; similarly, let... This represents the transformation of L at timestamp t into the pose in map coordinate system W. Finally, the method curve and position (n) are defined. w ,p w ) Parametrically define a specific plane in the map coordinate system. The Γ operation is performed via... The image coordinates are converted to map coordinates, and the intersection of x and the plane (n,p) is found, thus defining the geometric loss formula (Formula 2).

[0092]

[0093] The feature normal of the plane is derived from the point cloud plane where the projected image ray intersects it. The association function from the image trajectory to the point cloud map is established through the loss function based on the initial estimate of the camera to the lidar attitude and camera characteristics (Equation 3).

[0094]

[0095] Single camera gradually optimized

[0096] First, an external optimization loop is used for each camera-to-LiDAR location to improve the initial estimate, reject outlier trajectories, and estimate the feature point locations for multi-camera corresponding detection. In each iteration, step S3 is executed to remove trajectories with low correlation and use RANSAC to find the corresponding external parameters of the trajectory, making the correlation threshold between the trajectory and the map tighter. This process is repeated, adding new trajectories to the initial trajectory set, until the external parameters converge.

[0097] Multi-camera joint optimization

[0098] Simultaneously, the two camera estimates are projected onto the laser map. Since the positions and field of view of different cameras are different, more feature matching can be obtained than with a single camera. The parameters obtained in step S4 are used to optimize the parameters, and the optimization steps in step S4 are repeated to obtain the final calibration parameters from the two cameras to the laser. The final calibration parameters are then used to automatically replace the calibration parameters in the system, thus completing the automatic update of the calibration parameters.

[0099] Example 2

[0100] An electronic device, the electronic device comprising:

[0101] The memory stores multiple instructions;

[0102] A processor that loads instructions from the memory to execute the steps in the automatic calibration method described in Embodiment 1.

[0103] Example 3

[0104] A computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the automatic calibration method described in Embodiment 1.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for automatic calibration of multiple sensors in a train, characterized in that, Includes the following steps: Step S1: Before calibration begins, save the image data acquired by the camera and the laser point cloud data acquired by the LiDAR at a set time. Step S2: Preprocess the camera image data and lidar point cloud data saved in step S1 to obtain the camera feature trajectory and the surface equation of the lidar point cloud map. Step S3: Establish a correlation function between the feature trajectory and the point cloud map obtained in step S2 by associating them through a loss function; Step S4: Optimize the individual camera and LiDAR based on the correlation function between the feature trajectory and the point cloud map obtained in Step S3; Step S5: Optimize multiple cameras and LiDAR together using the method in step S4 to obtain optimized calibration parameters for multiple cameras and LiDAR; Step S6: Replace the original calibration parameters with the optimized calibration parameters obtained in step S5; Step S3 includes the following steps: set up A function representing the image pixels projected onto the i-th camera from a 3D point centered at the camera; make ∈SE(3) represents the pose of the i-th camera, which is transformed from a camera-centered coordinate system to a laser-centered coordinate system L; set up ∈SE(3) means transforming L at timestamp t into the pose of map coordinate system W; Method lines and positions Parametrically define a specific plane in the map coordinate system; Operation successful ( ) The image coordinates are converted to map coordinates, and the intersection of x and the plane (n, p) is found. Geometric loss formula 2 is then defined. The feature normals of the plane are derived from the point cloud plane where the projected image rays intersect. Based on the initial estimates of the camera-to-LiDAR pose and camera characteristics, a loss function is used to establish the association function between the feature trajectory and the point cloud map, i.e., Equation 3: 。 2. The automatic calibration method for multiple sensors in a train according to claim 1, characterized in that, In step S2, feature trajectories are first extracted for each camera using the SceneRF method; then, the ICP algorithm is used to accumulate and superimpose laser point cloud data over a set time period, and the superimposed laser point cloud data is initialized by the self-motion estimation of the IMU, moving object point clouds are removed, ground point clouds are removed from the accumulated point cloud data by the ground removal method, and Poisson surface reconstruction is performed on the non-ground point clouds to obtain the point cloud surface.

3. The automatic calibration method for multiple sensors in a train according to claim 2, characterized in that, Feature trajectory extraction for each camera using the SceneRF method includes the following steps: Feature detection was performed using the SIFT algorithm. Feature matching, matching and track building, pairwise matching of image pairs using Euclidean distance, a successful match is considered when the distance is less than a set threshold, false matches are removed, after all matching tracks are found, an image connectivity graph is constructed, which contains the nodes of each image and the edges of images with common tracks; Motion structure recovery yields estimated camera parameters and scene geometry information, i.e., sparse 3D point clouds.

4. The automatic calibration method for multiple sensors in a train according to claim 1, characterized in that, The optimization method for a single camera and LiDAR is as follows: First, an external optimization loop from single camera to LiDAR is used to improve the initial estimate, reject abnormal trajectories, and estimate the feature point positions for multi-camera corresponding detection. In each iteration, step S3 is executed to remove trajectories with low correlation and use RANSAC to find the corresponding extrinsic parameters of the trajectories, making the correlation threshold between the trajectories and the map tighter. This process is repeated to add new trajectories to the initial trajectory set until the extrinsic parameters converge.

5. The automatic calibration method for multiple sensors in a train according to claim 1, characterized in that, The method for optimizing multiple cameras and LiDAR together is as follows: simultaneously project the estimates of two cameras onto the LiDAR map; then use the parameters obtained from step S4 to repeat the optimization steps of step S4 to obtain the final calibration parameters of the two cameras to the LiDAR.

6. The automatic calibration method for multiple sensors in a train according to claim 1, characterized in that, In step S1, the time is set to 10 seconds.

7. The automatic calibration method for multiple sensors in a train according to claim 1, characterized in that, Set the automatic calibration time interval, and the above-mentioned automatic calibration method for multiple train sensors will be executed automatically at the set time.

8. An electronic device, characterized in that, The electronic device includes: The memory stores multiple instructions; A processor that loads instructions from the memory to perform the steps in the automatic calibration method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program performs the steps of the automatic calibration method according to any one of claims 1 to 7.

Images