A train positioning method, device, and equipment based on multi-source information fusion

Abstract

A train positioning method, apparatus, and device based on multi-source information fusion, relating to the field of railway positioning technology, includes: acquiring lidar data, IMU data, and train image data; fusing the lidar data and IMU data to obtain a laser inertial odometer trajectory, and fusing the IMU data and train image data to obtain a visual inertial odometer trajectory; identifying kilometer markers in the train image data and calculating the distance between the kilometer markers and the train to obtain the train's absolute position; correcting the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the train's absolute position; performing degradation detection, and fusing the corrected laser inertial odometer trajectory and the visual inertial odometer trajectory using a Kalman filter to calculate the train's real-time position and positioning information interval; this method, based on kilometer marker correction of the laser inertial odometer and visual inertial odometer trajectory and fusion of positioning results, achieves higher positioning accuracy than existing methods and exhibits good robustness.

Description

Technical Field

[0001] This invention relates to the field of railway positioning technology. Background Technology

[0002] With the large-scale construction of high-speed railways and the significant development of high-speed train control technology in recent years, the CTCS-4 train control system has been adopted for train positioning, planning, and control. High-speed train positioning primarily relies on transponders extensively deployed along the railway line. When a train passes over a transponder, it receives the electromagnetic signal emitted by the transponder, and the train control system deciphers this signal to obtain information such as basic line parameters, line speed, positioning data, and station routes. To reduce reliance on transponders, existing research largely focuses on integrating other sensors to achieve train positioning when GPS signals are available.

[0003] Current positioning strategies still rely on speed sensors to estimate distance, combined with trackside systems such as transponders and track circuits to correct position. This not only fails to meet the demands of real-time positioning but also requires significant investment in construction and subsequent maintenance. Specifically, existing solutions require GNSS to provide absolute position to eliminate accumulated errors. In tunnels and deep valleys where satellite positioning signals are lost for extended periods, this poses a severe challenge to the positioning system, and obtaining globally consistent maps is difficult. Furthermore, traditional train positioning technology's reliance on trackside electrical equipment leads to excessively high costs and maintenance difficulties in special operating environments. Laying expensive transponders further increases the cost of train positioning, and traditional positioning methods exhibit poor robustness in complex environments.

[0004] Therefore, how to provide a low-cost and robust train positioning method has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a train positioning method, apparatus, and device based on multi-source information fusion. It utilizes YOLO and OCR technologies to identify and detect digital information on kilometer markers, and then corrects the trajectories of laser inertial odometers and visual inertial odometers based on the kilometer markers and fuses the positioning results. This method has higher positioning accuracy than existing methods, meets the requirements for high-speed real-time train positioning, and has good robustness.

[0006] A train positioning method based on multi-source information fusion includes:

[0007] S1. Acquire lidar data, IMU data, and train image data;

[0008] S2. The laser inertial odometry trajectory is obtained by fusing the lidar data and IMU data, and the visual inertial odometry trajectory is obtained by fusing the IMU data and train image data.

[0009] S3. Identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining the high-precision railway network map;

[0010] S4. Correct the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the absolute position of the train;

[0011] S5. Perform degradation detection, and calculate the real-time position of the train and the fixed position signal interval by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter.

[0012] Furthermore, in step S2, the lidar data and IMU data are fused based on the error state Kalman filter, and the IMU data and train image data are fused based on the VINS-fusion framework.

[0013] Further, identifying kilometer markers in the train image data and calculating the distance between the kilometer markers and the train includes:

[0014] The kilometer markers are detected and identified using a YOLOv5 network, and digital recognition of the kilometer markers is performed using the PaddleOCR method to obtain the kilometer markers and their digital information.

[0015] Based on the coordinate information of the Bounding Box, the point cloud corresponding to the Bounding Box region is obtained by registering the image and point cloud with the calibrated lidar and camera extrinsic parameters.

[0016] Points in the point cloud with a reflection intensity greater than a preset threshold are selected, and a planar point cloud is fitted based on the RANSAC algorithm. The planar point cloud is then used to determine whether it corresponds to a kilometer marker based on the actual kilometer marker size. The centroid coordinates of the planar point cloud represent the kilometer marker position.

[0017] Based on the train speed at the time of point cloud acquisition, the distance to be compensated is extrapolated at a constant speed, and the distance between the compensated kilometer marker and the train is obtained.

[0018] Furthermore, the kilometer marker is an aluminum plate with a thickness of 1.2mm and a size of 50cm×40cm, and the surface of the kilometer marker is covered with a reflective film.

[0019] Furthermore, in the YOLOv5 network:

[0020] The input uses Mosaic data augmentation, employing random scaling, random cropping, and random arrangement for stitching, and uses adaptive image scaling in the Anchor part;

[0021] The backbone uses the Focus module for fast downsampling, and also employs the CSPBottleneck module with 3 convolutions and the SPP module;

[0022] The neck section uses a structure combining FPN and PAN, the bounding box regression loss function at the output end adopts CIOU_Loss, and the detection layer uses the Sigmoid activation function.

[0023] Further, step S4 includes:

[0024] S41. Obtain the four vertices of the kilometer marker and combine them with the kilometer marker's digital information to obtain the world coordinates of the vertices;

[0025] S42. Convert the world coordinates to coordinates in the camera coordinate system and calculate the reprojection error of the vertex;

[0026] S43. When the kilometer marker is detected for the first time, the set of feature points matched between the initial frame and the last frame in which the first kilometer marker appears is denoted as set. All IMU data is denoted as set . Let L be the set of all image frames that detect kilometer marker N. Construct the correction and optimization objective function as follows:

[0027]

[0028] Where, r p -H p χ represents the prior information after all sliding windows are marginalized. This represents the residual of the IMU inertial constraints between all two frames. Let (l,k) represent the reprojection error of the visual landmark, and (l,k) represent the l-th image feature observed in the k-th image frame. ρ represents the reprojection error of the kilometer marker vertex, and ρ represents the robust kernel function.

[0029] When multiple kilometer markers are detected, an optimization problem is constructed between the two most recent kilometer markers for global optimization.

[0030] S44. Optimize the pose of the laser point cloud and image keyframes between two kilometer markers, where the kilometer marker pose factor is: ΔT kmp =(T i T i j ) T T j ;

[0031] Among them, T j To measure the pose of the j-th kilometer marker stored in the electronic map, T i jThe pose of the j-th kilometer marker relative to the train is observed in the i-th frame of the point cloud.

[0032] Further, step S5 includes:

[0033] S51. Define laser degradation factor and visual degradation factor to perform degradation detection on laser inertial and visual inertial odometer trajectories, and calculate the fusion weight of the laser inertial odometer trajectory and visual inertial odometer trajectory based on the detection results.

[0034] S52. Calculate the degenerate covariance matrix based on the fusion weights, expressed by the following formula:

[0035]

[0036] Where, ρ L For laser fusion weights, ρ V For visual fusion weights, Σ L Σ represents the covariance of the train position output by the laser inertial odometry. V This represents the covariance corresponding to the train position output by the visual inertial odometry. Considering the degenerate covariance matrix for laser inertial odometry, Considering the degraded covariance matrix for visual inertial odometry;

[0037] S53. Calculate the Kalman filter gain based on the degenerate covariance matrix, and then obtain the displacement at each time step after fusion positioning;

[0038] S54. The relative displacements at adjacent time points are accumulated to obtain the one-dimensional travel distance, which is expressed by the following formula:

[0039]

[0040] Among them, L t The real-time position of the train at time t. The displacement after fusion and positioning at time K+1;

[0041] S55. Obtain the prior standard deviation of the train's real-time position using the edge-forming method in graph optimization. Thus, the train positioning information interval is obtained.

[0042] A train positioning device based on multi-source information fusion includes:

[0043] The data acquisition module is used to acquire lidar data, IMU data, and train image data;

[0044] The train positioning odometer module is used to fuse the lidar data and IMU data to obtain the laser inertial odometer trajectory, and to fuse the IMU data and train image data to obtain the visual inertial odometer trajectory.

[0045] The kilometer marker identification and positioning module is used to identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining it with a high-precision railway network map;

[0046] The trajectory correction module is used to correct the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the absolute position of the train.

[0047] The joint estimation module is used to perform degradation detection and calculate the real-time position and positioning information interval of the train by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter.

[0048] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0049] An electronic device includes a processor and a storage device, the storage device storing a plurality of instructions, the processor being configured to read the plurality of instructions from the storage device and execute the method described above.

[0050] The train positioning method, apparatus, and equipment based on multi-source information fusion provided by this invention have at least the following beneficial effects:

[0051] To address the problems of high costs and maintenance difficulties in special operating environments caused by the reliance on trackside electrical equipment in traditional train positioning technologies, this invention proposes a positioning system that fuses trackside kilometer marker location information using laser inertial odometers and visual inertial odometers. This system achieves high-precision real-time autonomous train positioning in complex environments. Furthermore, by fusing laser inertial odometer and visual inertial odometer data within a Kalman filter framework and using degradation scene detection, the system exhibits strong robustness. The kilometer marker-based odometer positioning trajectory correction method achieves positioning accuracy far exceeding that of existing train positioning systems, ensuring long-term stable train positioning. The use of kilometer markers can significantly reduce or even eliminate the need for expensive transponders. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1A flowchart of one embodiment of the train positioning method based on multi-source information fusion provided by the present invention;

[0054] Figure 2 Design drawings for kilometer markers;

[0055] Figure 3 This is a diagram of the YOLOv5 network structure.

[0056] Figure 4 Here is a flowchart of the image-point cloud joint ranging process;

[0057] Figure 5 A schematic diagram for eliminating accumulated errors in kilometer markers;

[0058] Figure 6 Method for determining the vertex of a kilometer marker;

[0059] Figure 7 This is a schematic diagram of the initial repositioning during the calibration process;

[0060] Figure 8 This is a schematic diagram illustrating repositioning within a two-kilometer marker interval during multiple checks in the calibration process;

[0061] Figure 9 A factor diagram for laser inertial odometry;

[0062] Figure 10 The results of kilometer marker information identification under normal operating conditions;

[0063] Figure 11 The results of kilometer marker information recognition in low-light conditions;

[0064] Figure 12 The results of kilometer marker identification within the tunnel section;

[0065] Figure 13 The curve showing the change in the confidence interval of the localization result;

[0066] Figure 14 This is a curve diagram of the positioning trajectory. Detailed Implementation

[0067] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0068] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0069] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0072] Example 1:

[0073] See Figure 1 In some embodiments, a train positioning method based on multi-source information fusion is provided, characterized by comprising:

[0074] S1. Acquire lidar data, IMU data, and train image data;

[0075] S2. The laser inertial odometry trajectory is obtained by fusing the lidar data and IMU data, and the visual inertial odometry trajectory is obtained by fusing the IMU data and train image data.

[0076] S3. Identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining the high-precision railway network map;

[0077] S4. Correct the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the absolute position of the train;

[0078] S5. Perform degradation detection, and calculate the real-time position of the train and the fixed position signal interval by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter.

[0079] This method addresses the need to reduce trackside equipment installation and addresses typical scenarios with limited satellite signals, such as tunnels and deep valleys. It employs a multi-source information fusion positioning technology, utilizing lidar, cameras, inertial measurement units (IMUs), and kilometer markers, to achieve highly reliable determination of the train's position and direction of travel on the track. This meets the requirements for safe train positioning, improves train operating efficiency, and enhances adaptability to complex operating environments. Typical train operating environments include tunnels, bridges, and open areas; some trains may also operate at night. While lidar is unaffected by lighting and can accurately perceive the surrounding environment, its sparse information leads to rapid degradation in unstructured scenarios. Visual odometry, on the other hand, offers rich environmental information, high frame rates, and low cost, but its dependence on initialization and sensitivity to lighting cause system instability. Therefore, even in the same scenario, the positioning accuracy of lidar-based inertial odometry and visual inertial odometry differs, and they may fail in certain specific scenarios. However, they are highly complementary. Therefore, this paper combines the positioning results of both odometry systems to achieve real-time, high-precision train position information across all time periods and scenarios.

[0080] Specifically, in step S2, the lidar data and IMU data are fused based on the error state Kalman filter, and the IMU data and train image data are fused based on the VINS-fusion algorithm.

[0081] First, the laser inertial odometry is explained. To robustly estimate pose in railway scenarios, this embodiment employs an Error State Kalman Filter (ESKF) to fuse LiDAR (Laser Detecting and Ranging) and IMU (Inertial Measurement Unit) measurement data. Compared to traditional Kalman filtering methods, ESKF offers advantages such as minimizing parameter rotation, numerical stability, and small, negligible second-order state variables to reduce computational complexity.

[0082] Considering the status of the train positioning system:

[0083]

[0084] In the formula, The parameters are, in order: IMU position, rotation matrix, LiDAR-IMU rotation and translation extrinsic parameters, IMU velocity, IMU gyroscope and accelerometer zero bias, and gravity vector.

[0085] Define error states:

[0086] x t =x + δx; (2)

[0087] In the formula, xt Let x be the actual state, and let x be the nominal state, i.e., the state without considering noise.

[0088] The overall ESKF process is as follows: When IMU measurement data arrives, it is integrated and added to the nominal state variable. Due to noise, the result will drift quickly, so the error component is used as the error state. During the process, the nominal state is recursively derived by integrating the IMU data, while the error state is affected by Gaussian noise and increases. Its specific value is described by the mean and covariance of the ESKF error state. Furthermore, the ESKF update process is completed by lidar observations. A measurement model is constructed using lidar point measurements, updating the posterior mean and covariance of the error state. Then, the error state is added to the nominal state, thus completing one prediction-update cycle.

[0089] Based on kinematic derivation, the following motion model equations can be established:

[0090]

[0091] In the formula, ω m ,a m Let (a) represent the IMU measurement value, and (a)^ represent the antisymmetric matrix of vector a.

[0092] The state recursion equation in discrete time can be easily derived from equation (3), which will not be elaborated further. Therefore, the nominal state can be recursively derived using IMU measurements. Furthermore, by treating each state on the manifold, the error state differential equation becomes:

[0093]

[0094] Therefore, the discrete-time error state motion equation is:

[0095]

[0096] The above formula can be simplified as follows:

[0097] δx k+1 =f(δx) k )+w,w~Ν(0,Q); (6)

[0098] To propagate the covariance, equation (6) is linearized.

[0099] δx k+1 =F δx δx k +w; (7)

[0100] In the formula, F δx The Jacobian matrix of expression (6) relative to δx.

[0101] Therefore, the ESKF prediction process includes the prediction of the nominal state and the prediction of the error state, that is:

[0102]

[0103] During point cloud registration, point-to-surface residuals are constructed and used as measurements for ESKF updates.

[0104]

[0105] In the formula, u j Represents a point in the world coordinate system W p j The normal vector of the local plane corresponding to the point. W q represents a point in the local plane corresponding to this point.

[0106] Generally, the above formula can be expressed as:

[0107] z = h(x) + v, v ~ N(0, V); (10)

[0108] Therefore, the ESKF update process is as follows:

[0109]

[0110] In this case, the Kalman gain K, compared to the traditional form, only requires inverting the matrix in the state dimension rather than the measurement dimension.

[0111] Next, visual inertial odometry will be explained. The VINS-fusion algorithm is a type of visual inertial odometry calculation method, mainly used for high-precision positioning and trajectory reconstruction of mobile devices such as drones and robots in complex environments. Its principle can be simply described as follows: using a visual camera to acquire environmental images, using an inertial measurement unit (IMU) to acquire device motion information, and then fusing the two to achieve estimation and tracking of the device's position and attitude.

[0112] In summary, the core idea of ​​the VINS-fusion algorithm is to use inertial data provided by the IMU and camera image data for collaborative computation to continuously optimize the estimation results, so as to achieve more accurate and robust mobile device positioning.

[0113] Based on VINS-fusion fusion of images and IMU measurements to estimate train status in real time, the train localization problem is modeled as a maximum likelihood estimation problem.

[0114] (x,y) * =arg max P(z|x,y); (12)

[0115] In the formula, x and y represent the camera state and the pose of the landmark, and z represents the camera's measurement of the landmark.

[0116] The front end extracts FAST corner points and performs pose estimation based on the LK optical flow method. During the operation of the front end odometry, errors accumulate, requiring the back end to establish a larger-scale optimization problem to eliminate these errors. Equation (12) is transformed into a nonlinear least squares problem as shown in Equation (13). Finally, the global optimization of the back end is achieved through nonlinear optimization, which can obtain the optimal solution for the camera pose. The error term includes visual point feature reprojection error, IMU error, and prior information after marginalization in the sliding window:

[0117]

[0118] In the formula, r p -H p χ represents the prior information after marginalization. This represents the residual of the IMU inertial constraint between two frames within the sliding window. This represents the reprojection error of visual landmarks. Let (l,k) represent all IMU data within the sliding window, and let (l,k) represent the l-th image feature observed in the k-th image frame.

[0119] The algorithm's backend constructs a sliding window factor map to optimize camera and landmark poses. Visual factors are constructed from reprojection errors to constrain the poses of multiple cameras sharing common observation points. IMU pre-integration values ​​and observation differences are used as IMU factors to constrain the states between adjacent frames. Previous measurements removed from the sliding window are marginalized as a prior term and added to the factor map. Furthermore, to ensure real-time performance, a multi-threaded programming approach is used to create two separate threads responsible for the frontend and backend.

[0120] In step S3, the kilometer marker digit detection employs PaddleOCR technology, an OCR (Optical Character Recognition) technology based on the deep learning framework PaddlePaddle. It features ultra-lightweight design, small model size, and ease of deployment on mobile and server-side devices. To accurately obtain the distance to the identified kilometer markers, an image-point cloud joint kilometer marker ranging method is proposed: after identifying the kilometer marker, the image and point cloud are registered to obtain the point cloud corresponding to the kilometer marker anchor frame region, and then the distance between the kilometer marker and the camera is measured. The specific process is as follows: Figure 4 .

[0121] Identifying kilometer markers in the train image data and calculating the distance between the kilometer markers and the train includes:

[0122] S31. The kilometer markers are detected and identified using a YOLOv5 network, and the kilometer markers are digitally identified using the PaddleOCR method to obtain the kilometer markers and their digital information.

[0123] S32. Based on the kilometer marker's digital information, return the Bounding Box coordinate information, and use the calibrated lidar and camera extrinsic parameters to register the image and point cloud to obtain the corresponding point cloud of the Bounding Box region;

[0124] S33. Filter out points in the point cloud whose reflection intensity is greater than a preset threshold, fit the planar point cloud based on the RANSAC algorithm, determine whether the planar point cloud corresponds to the kilometer marker based on the actual kilometer marker size, and use the centroid coordinates of the planar point cloud to represent the kilometer marker position.

[0125] S34. Based on the train speed at the point cloud acquisition time, extrapolate the distance to be compensated to obtain the distance between the kilometer marker and the train after compensation.

[0126] After identifying the kilometer marker and its digital information, the bounding box coordinates are returned. Using calibrated LiDAR-camera extrinsic parameters to register the image and point cloud, the corresponding point cloud for the bounding box region is obtained. Points with reflection intensity greater than a certain threshold are selected, and a plane is fitted using RANSAC. Further, based on the actual kilometer marker size, it is determined whether the planar point cloud corresponds to the kilometer marker, with the centroid coordinates of the planar point cloud representing the kilometer marker position. Simultaneously, considering the time difference between image frames and point cloud frames, as well as algorithm processing time, the output kilometer marker distance needs to compensate for the train's motion during this period. The required compensation distance is extrapolated from the train's constant speed at the point cloud acquisition time. Finally, the kilometer marker distance and confidence interval are output.

[0127] As a preferred implementation method, the kilometer marker used in this embodiment is as follows: Figure 2 As shown, the kilometer marker is a 1.2mm thick aluminum plate, measuring 50cm × 40cm, with a reflective film affixed to its surface. The numeral font used is Arial, and the high-quality reflective film on the marker surface ensures good reflection of auxiliary light sources in low-light conditions. The position results obtained by odometers inevitably accumulate errors over time. In applications such as autonomous driving and drone self-positioning, loop closure detection is commonly used to eliminate these errors. However, in railway operations, loop closure detection is typically not feasible. Therefore, this embodiment proposes a novel, low-cost trackside kilometer marker to replace existing meaningless stone kilometer markers. It uses deep learning methods to identify kilometer marker information and obtain the current accurate train position, providing a foundation for subsequent information fusion to eliminate accumulated errors.

[0128] As a preferred implementation, the kilometer marker identification section is based on YOLOv5s with relevant improvements. The network structure of YOLOv5 is as follows: Figure 3As shown, the network is mainly divided into four parts: input, backbone, neck, and output. In the YOLOv5 network: the input uses Mosaic data augmentation, employing random scaling, cropping, and arrangement for data stitching, and adaptive image scaling is used in the anchor part; the backbone uses a focus module for fast downsampling, and also employs a CSPBottleneck module with 3 convolutions and an SPP module; the neck uses a structure combining FPN (elemental pyramid networks) and PAN (path aggregation network); the output uses CIOU_Loss as the bounding box regression loss function, and the detection layer uses the Sigmoid activation function. CIOU_Loss is a loss function used to calculate the distance between two bounding boxes in object detection.

[0129] During odometer operation, each frame of input point cloud and image is detected. If a kilometer marker exists in the current frame and is successfully identified, its precise location is obtained based on the parsed digital information and the electronic map. After the kilometer marker disappears from the field of view, the system activates its global optimization function, incorporating the kilometer marker location information as a constraint into the objective function of nonlinear optimization. This constructs a large-scale optimization problem to eliminate accumulated errors and build a globally consistent trajectory and map. Figure 5 As shown, the positioning error gradually increases with train operation. The accumulated positioning error can be effectively eliminated by using the absolute position information of the kilometer marker.

[0130] In a preferred embodiment, step S4 includes:

[0131] S41. Obtain the four vertices of the kilometer marker and combine them with the kilometer marker's digital information to obtain the world coordinates of the vertices;

[0132] S42. Convert the world coordinates to coordinates in the camera coordinate system and calculate the reprojection error of the vertex;

[0133] S43. When the kilometer marker is detected for the first time, the set of feature points matched between the initial frame and the last frame in which the first kilometer marker appears is denoted as set. All IMU data is denoted as set . Let L be the set of all image frames that detect kilometer marker N. Construct the correction and optimization objective function as follows:

[0134]

[0135] Where, r p -H p χ represents the prior information after all sliding windows are marginalized. This represents the residual of the IMU inertial constraints between all two frames. Let (l,k) represent the reprojection error of the visual landmark, and (l,k) represent the l-th image feature observed in the k-th image frame. This represents the reprojection error of the kilometer marker's vertex.

[0136] When multiple kilometer markers are detected, an optimization problem is constructed between the two most recent kilometer markers for global optimization.

[0137] S44. Optimize the pose of the laser point cloud and image keyframes between two kilometer markers, where the kilometer marker pose factor is: ΔT kmp =(T i T i j ) T T j ;

[0138] Among them, T j To measure the pose of the j-th kilometer marker stored in the electronic map, T i j The pose of the j-th kilometer marker relative to the train is observed in the i-th frame of the point cloud.

[0139] In step S41, this embodiment obtains the intersection points of the four sides of the detected rectangular kilometer marker. The vertex determination method is as follows: Figure 6 As shown, a1 and a2 are horizontal edge lines, b1 and b2 are vertical edge lines, and A ij This represents the kilometer marker vertex obtained by the intersection of four edges.

[0140] In step S42, based on the digital information recognition results and combined with the known electronic map, the coordinates of the kilometer marker vertex in the world coordinate system are obtained. Where N represents the kilometer marker's numerical index, and j = 1, 2, 3, 4 represent the four vertices of the rectangular kilometer marker. In the image sequence, kilometer marker N was observed in a total of M frames, where the coordinates of the kilometer marker's vertices detected in the i-th frame are... At this moment, the coordinates of the vertex of kilometer marker N in the camera coordinate system at time i are:

[0141]

[0142] In the formula, The transformation matrix representing the transformation from world coordinates to the IMU coordinate system at time i is obtained from the odometry system positioning results;

[0143] This represents the transformation matrix from the IMU coordinate system of the multi-sensor system to the camera coordinate system, obtained from the joint calibration results of the sensor system.

[0144] The specific form of the reprojection error of the kilometer marker vertex is shown in equation (15):

[0145]

[0146] In step S43, when the kilometer marker is detected for the first time, such as Figure 7 As shown, the set of feature points matched between the initial frame of the positioning system and the last frame where the first kilometer marker appears is set as... The complete IMU dataset is Let L be the set of all image frames that detected kilometer marker N. Construct the optimization objective function for relocalization:

[0147]

[0148] When multiple kilometer markers are detected, an optimization problem is constructed only between the two most recent kilometer markers for global optimization, such as... Figure 8 As shown, the relocation optimization objective function remains Equation (16), and at this time the set This represents the set of feature points matched between the last frame of the current kilometer marker and the first frame of the previous kilometer marker. This represents all IMU data within the interval, and set L is the set of all image frames that detected the current kilometer marker N and the previous kilometer marker N-1.

[0149] In step S44, to ensure real-time performance, a parallel thread is constructed to optimize and correct the global trajectory. At the same time, to control the optimization scale, the optimization is only performed on the pose of the laser point cloud and image keyframes between the two kilometer markers. The image keyframes are selected based on time and feature quantity, and the laser point cloud keyframes are selected based on appropriate thresholds set according to time difference and spatial position difference.

[0150] Build such a structure at the back end of the laser inertial odometry system. Figure 9 The factor plot shown is constructed from the pose estimated by ESKF in step S2:

[0151] ΔT odo =T i T T i+1 (17)

[0152] In the formula, T i,i+1 T represents the pose transformation of the laser point cloud keyframes from frame i to frame i+1. i This represents the pose of the i-th laser point cloud keyframe.

[0153] The j-th kilometer marker is observed in the point cloud of the i-th frame, and the kilometer marker pose T stored in the electronic map is measured. j The kilometer marker relative to the train's pose T, measured by lidar. ij Then the kilometer marker factor is:

[0154] ΔT kmp =(T i T i j ) T T j (18)

[0155] As a preferred implementation, step S5 includes:

[0156] S51. Define laser degradation factor and visual degradation factor to perform degradation detection on laser inertial and visual inertial odometer trajectories, and calculate the fusion weight of the laser inertial odometer trajectory and visual inertial odometer trajectory based on the detection results.

[0157] S52. Calculate the degenerate covariance matrix based on the fusion weights, expressed by the following formula:

[0158]

[0159] Where, ρ L For laser fusion weights, ρ V For visual fusion weights, Σ L Σ represents the covariance of the train position output by the laser inertial odometry. V This represents the covariance corresponding to the train position output by the visual inertial odometry. Considering the degenerate covariance matrix for laser inertial odometry, Considering the degraded covariance matrix for visual inertial odometry;

[0160] S53. Calculate the Kalman filter gain based on the degenerate covariance matrix, and then obtain the displacement at each time step after fusion positioning;

[0161] S54. The relative displacements at adjacent time points are accumulated to obtain the one-dimensional travel distance, which is expressed by the following formula:

[0162]

[0163] Among them, L t The real-time position of the train at time t. The displacement after fusion and positioning at time K+1;

[0164] S55. Obtain the prior standard deviation of the train's real-time position using the edge-forming method in graph optimization. Thus, the train positioning information interval is obtained.

[0165] In step S51, sensors will exhibit degradation in special scenarios such as tunnels, bridges, and areas with sparse textures. LiDAR will fail to provide adequate constraints in scenarios with sparse structural features, leading to degradation issues that are difficult to correct using LiDAR alone. Visual-inertial odometry is prone to failure in scenarios with drastic lighting changes and a lack of scene texture features. Therefore, combining position estimation with effective identification of degraded scenarios is more robust and accurate.

[0166] For the degradation problem of lidar, a degradation factor is used to judge the degradation status. For the linear optimization problem represented by equation (19)

[0167]

[0168] The laser degradation factor is defined as follows:

[0169] D L =λ min +1; (20)

[0170] In the formula, λ min A represents T The smallest eigenvalue of A.

[0171] When D L When the value is less than a certain threshold, a degradation scenario can be identified, and the direction of the most severe degradation is the direction of the eigenvector corresponding to the smallest eigenvalue. When degradation occurs, the demapping method is used to mitigate the impact of degradation. The optimal prediction value of the solution to equation (19) is x. p The solution obtained by solving equation (19) is x. u Then, in the direction of degradation, only x is used p The projection in that direction, while utilizing x in the direction with good conditions. u The projection, and only x is updated during iterative updates. u The final solution is a linear combination of the two.

[0172] x f =x′ p +x′ u ; (twenty one)

[0173] For the visual localization degradation problem, when the number of feature points extracted from multiple consecutive frames of images is n... f If the number of optical flow tracking feature points is less than a certain threshold, or the number of feature points is less than the threshold, and considering that the train moves at a near-uniform straight speed during operation, visual localization is considered unreliable or has failed when the estimated pose change ΔT between adjacent frames is too large. Define the visual degradation factor:

[0174] D V =f(n) f ,ΔT); (22)

[0175] When D V When the value is below a certain threshold, the visual inertial odometry is reinitialized using laser inertial odometry information.

[0176] Based on the completion of the real-time positioning system using laser inertial odometer and visual inertial odometer, and the identification of railway trackside kilometer markers, it is necessary to fuse position information to obtain the final positioning result. Compared with single-sensor odometers, the multi-sensor tightly coupled method significantly improves accuracy and robustness, but it has a large computational load, and the tightly coupled scheme is susceptible to sensor failure in degraded scenarios. The loosely coupled method, on the other hand, fuses the pose results output by each sensor (subsystem). When some sensor data fails or degrades, its pose weight can be reduced or ignored, thus ensuring the stability and accuracy of the fusion. Therefore, a loosely coupled framework is used, employing a Kalman filter to fuse LIO and VIO information.

[0177] Consider the LIO output positioning results:

[0178] x L ~N(p L ,Σ L );(twenty three)

[0179] In the formula, p L Indicates the train position output by LIO, Σ L This represents the covariance corresponding to the train's position.

[0180] Similarly, the VIO output is:

[0181] x V ~N(p V ,Σ V );(twenty four)

[0182] Where, Σ V p represents the covariance of the train position output by the visual inertial odometry. V The train position is output by VIO.

[0183] When degradation or failure occurs, the fusion weight of the corresponding positioning results is reduced to fully utilize the information from each sensor and ensure positioning accuracy and stability. The fusion weight factors for LIO and VIO positioning results are defined as follows:

[0184]

[0185]

[0186] Since degradation scenarios have a more severe impact on VIO than LIO, the fusion weighting factors for the two differ, taking the degree of degradation into account in the covariance matrix:

[0187]

[0188] The above localization results are then fused within the Kalman filter framework:

[0189]

[0190] In the formula, K is the Kalman filter gain;

[0191] The train control system only requires one-dimensional travel distance. Since both LIO and VIO systems can output positioning results at a frequency of at least 10Hz, the sum of the relative displacements at adjacent moments is used as the one-dimensional travel distance.

[0192]

[0193] Furthermore, due to the high reliability requirements of train operation, a corresponding confidence interval needs to be provided along with the positioning result to reduce the possibility of accidents caused by positioning errors. The prior standard deviation of the current position is obtained using the marginalization method in graph optimization. Therefore, the train positioning information interval is as follows:

[0194]

[0195] To further verify the effectiveness of the method provided in this embodiment, relevant experimental verifications are described below.

[0196] To improve image processing speed and meet the real-time requirements of train positioning, the hardware configuration for this experiment is as follows: NVIDIA Jetson AGX Orin development kit (12-core Arm Cortex-A78AE 64-bit CPU, 64GB memory), RoboSense RS-M1 LiDAR, ZED2 binocular camera and its built-in IMU, GNSS positioning module (providing positioning truth).

[0197] To verify the reliability of the railway trackside kilometer marker detection method based on the YOLOv5 model proposed in this embodiment, training and validation on a hardware platform are required. The experiments in this paper primarily used Ubuntu operating system, version 18.04, and the network was built using the PyTorch deep learning framework, with Python 3.8 as the programming language. The kilometer marker recognition and digit detection algorithms described in Section 3.2 were used to analyze the kilometer marker recognition results in the dataset. Figures 10-12 As shown. The method proposed in this embodiment achieves a detection accuracy of 95.2% and a recall rate of 99.7% when facing railway trackside kilometer marker images with different lighting, complex backgrounds, and varying shapes. At the same time, the lightweight network model supports deployment in vehicle-mounted embedded devices and ensures the real-time performance of the system.

[0198] To verify the effectiveness and reliability of the proposed method and to simulate the railway line environment as closely as possible, an instrument cart was used to collect data along a 120-meter-long forest trail. A kilometer marker was placed every 60 meters along the trail. The location trajectory and confidence interval changes are as follows: Figures 13-14 As shown in the figure, the horizontal axis of the confidence interval variation curve represents the keyframe index, the short solid line in the trajectory curve represents the actual trajectory obtained by GNSS, the dashed line represents the trajectory before positioning correction, and the solid line represents the trajectory after positioning correction.

[0199] Experimental results show that the method proposed in this embodiment can effectively eliminate the cumulative positioning error after the kilometer marker is identified, and the optimized positioning trajectory coincides with the real trajectory, thereby establishing a globally consistent trajectory and map to ensure accurate positioning of the train over a long period of time.

[0200] Example 2:

[0201] In some embodiments, a train positioning device based on multi-source information fusion is provided, comprising:

[0202] The data acquisition module is used to acquire lidar data, IMU data, and train image data;

[0203] The train positioning odometer module is used to fuse the lidar data and IMU data to obtain the laser inertial odometer trajectory, and to fuse the IMU data and train image data to obtain the visual inertial odometer trajectory.

[0204] The kilometer marker identification and positioning module is used to identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining it with a high-precision railway network map;

[0205] The trajectory correction module is used to correct the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the absolute position of the train.

[0206] The joint estimation module is used to perform degradation detection and calculate the real-time position and positioning information interval of the train by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter.

[0207] The train positioning odometer module comprises two sub-modules: Laser-Inertial Odometry (LIO) and Visual-Inertial Odometry (VIO). These two sub-modules complement each other, enabling robust real-time positioning and speed measurement across all scenarios. In particular, it solves the positioning challenges in tunnel sections, and the system can still function normally and recover quickly even if one odometer fails. The kilometer marker recognition and positioning module identifies trackside kilometer markers based on images and determines their distance relative to the train. It then combines this information with a high-precision railway network map to obtain the train's current accurate position. The trajectory correction module corrects the positioning trajectory of the LIO and VIO systems based on the absolute position information provided by the kilometer markers, eliminating accumulated errors and ensuring long-term, long-distance positioning accuracy. The train position joint estimation module combines the positioning results of LIO and VIO to estimate the train's real-time position and positioning information interval, fulfilling the positioning function requirements of the train operation control system.

[0208] Example 3:

[0209] In some embodiments, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0210] Example 4:

[0211] In some embodiments, an electronic device is provided, including a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions from the storage device and execute the method described above.

[0212] This invention provides a train positioning method, apparatus, and equipment based on multi-source information fusion. Addressing the problems of high costs and maintenance difficulties in special operating environments caused by the reliance on trackside electrical equipment in traditional train positioning technologies, this invention proposes a positioning system that fuses trackside kilometer marker position information using laser inertial odometers and visual inertial odometers. This achieves high-precision real-time autonomous train positioning in complex environments. Furthermore, by fusing laser inertial odometer and visual inertial odometer data within a Kalman filtering framework and degradation scene detection, the system exhibits strong robustness. The kilometer marker-based odometer positioning trajectory correction method achieves positioning accuracy far exceeding that of existing train positioning systems, ensuring long-term stable train positioning. The use of kilometer markers can significantly reduce or even eliminate the need for expensive transponders.

[0213] It should be understood that in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, such as microprocessors or any conventional processors.

[0214] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0215] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0216] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0217] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of the modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0219] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A train positioning method of multi-source information fusion, characterized in that, include: S1. Acquire lidar data, IMU data, and train image data; S2. The laser inertial odometry trajectory is obtained by fusing the lidar data and IMU data, and the visual inertial odometry trajectory is obtained by fusing the IMU data and train image data. S3. Identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining the high-precision railway network map; S4. Correct the laser inertial odometer trajectory and the visual inertial odometer trajectory based on the absolute position of the train; S5. Perform degradation detection, and calculate the real-time position of the train and the fixed position signal interval by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter. Step S4 includes: S41. Obtain the four vertices of the kilometer marker and combine them with the kilometer marker's digital information to obtain the world coordinates of the vertices; S42. Convert the world coordinates to coordinates in the camera coordinate system and calculate the reprojection error of the vertex; S43, when the milestone is detected for the first time, the set of feature points matched between the initial frame and the last frame before the first milestone appears is recorded as set , all IMU data is recorded as set , all image frames in which milestone N is detected are recorded as set L, and a rectification optimization objective function is constructed, which is represented as follows: ； where, represents all the prior information marginalized over all sliding windows, represents the residual of all the IMU inertial constraints between two frames, represents the re-projection error of visual landmarks, represents the image feature observed by the image frame, represents the vertex re-projection error of the milepost, represents the robust kernel function; When multiple kilometer markers are detected, an optimization problem is constructed between the two most recent kilometer markers for global optimization. S44, optimizing the laser point cloud and the image key frame pose between the two mile markers, wherein the mile marker pose factor is: ; in, To measure the pose of the j-th kilometer marker stored in the electronic map, The pose of the j-th kilometer marker relative to the train is observed in the i-th frame of the point cloud.

2. The method according to claim 1, characterized in that, In step S2, the lidar data and IMU data are fused based on the error state Kalman filter, and the IMU data and train image data are fused based on the VINS-fusion framework.

3. The method according to claim 1, characterized in that, Identifying kilometer markers in the train image data and calculating the distance between the kilometer markers and the train includes: The kilometer markers are detected and identified using a YOLOv5 network, and digital recognition of the kilometer markers is performed using the PaddleOCR method to obtain the kilometer markers and their digital information. Based on the coordinate information of the Bounding Box, the point cloud corresponding to the Bounding Box region is obtained by registering the image and point cloud with the calibrated lidar and camera extrinsic parameters. Points in the point cloud with a reflection intensity greater than a preset threshold are selected, and a planar point cloud is fitted based on the RANSAC algorithm. The planar point cloud is then used to determine whether it corresponds to a kilometer marker based on the actual kilometer marker size. The centroid coordinates of the planar point cloud represent the kilometer marker position. Based on the train speed at the time of point cloud acquisition, the distance to be compensated is extrapolated at a constant speed, and the distance between the compensated kilometer marker and the train is obtained.

4. The method according to claim 3, characterized in that, The kilometer marker is made of an aluminum plate with a thickness of 1.2mm and a size of 50cm×40cm. The surface of the kilometer marker is covered with a reflective film.

5. The method according to claim 3, characterized in that, In the YOLOv5 network: The input uses Mosaic data augmentation, employing random scaling, random cropping, and random arrangement for stitching, and uses adaptive image scaling in the Anchor part; The backbone uses the Focus module for fast downsampling, and also employs the CSPBottleneck module with 3 convolutions and the SPP module; The neck section uses a structure combining FPN and PAN, the bounding box regression loss function at the output end adopts CIOU_Loss, and the detection layer uses the Sigmoid activation function.

6. The method according to claim 1, characterized in that, Step S5 includes: S51. Define laser degradation factor and visual degradation factor to perform degradation detection on laser inertial and visual inertial odometer trajectories, and calculate the fusion weight of the laser inertial odometer trajectory and visual inertial odometer trajectory based on the detection results. S52. Calculate the degenerate covariance matrix based on the fusion weights, expressed by the following formula: ； in, For laser fusion weights, For visual fusion weights, This represents the covariance corresponding to the train position output by the laser inertial odometer. This represents the covariance corresponding to the train position output by the visual inertial odometer. Considering the degenerate covariance matrix of laser inertial odometry, Considering the degraded covariance matrix for visual inertial odometry; S53. Calculate the Kalman filter gain based on the degenerate covariance matrix, and then obtain the displacement at each time step after fusion positioning; S54. The relative displacements at adjacent time points are accumulated to obtain the one-dimensional travel distance, which is expressed by the following formula: ； in, The real-time position of the train at time t. The displacement after fusion and positioning at time K+1; S55. Obtain the prior standard deviation of the train's real-time position using the edge-forming method in graph optimization. Thus, the train positioning information interval is obtained. .

7. A train positioning device based on multi-source information fusion, characterized in that, The train positioning method using multi-source information fusion as described in any one of claims 1-6 includes: The data acquisition module is used to acquire lidar data, IMU data, and train image data; The train positioning odometer module is used to fuse the lidar data and IMU data to obtain the laser inertial odometer trajectory, and to fuse the IMU data and train image data to obtain the visual inertial odometer trajectory. The kilometer marker identification and positioning module is used to identify kilometer markers in the train image data and calculate the distance between the kilometer markers and the train, and obtain the absolute position of the train by combining it with a high-precision railway network map; A trajectory correction module is used to correct the trajectory of the laser inertial odometer and the visual inertial odometer based on the absolute position of the train. The joint estimation module is used to perform degradation detection and calculate the real-time position and positioning information interval of the train by fusing the corrected laser inertial odometer trajectory and visual inertial odometer trajectory through a Kalman filter.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

9. An electronic device comprising a processor and a storage device, characterized in that, The storage device contains multiple instructions, and the processor is used to read the multiple instructions in the storage device and execute the method as described in any one of claims 1-6.

Images