Railway track detection method and system based on vision and laser ranging
By employing a multi-sensor tightly coupled approach combining vision and laser ranging, a unified optimization estimation framework is constructed. By integrating vision, laser, and IMU data and introducing track geometry rules, the problems of low measurement efficiency, insufficient accuracy, and error accumulation in railway track inspection are solved, achieving high-precision and robust track state estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI CHINA RAILWAY ENG TECH SERVICE CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-02
AI Technical Summary
Existing railway track inspection technologies suffer from low measurement efficiency, insufficient accuracy, error accumulation, weak adaptability to different scenarios, and a lack of optimization mechanisms specifically for railway scenarios, making it difficult to meet the requirements of high-density operation and high precision.
A multi-sensor tightly coupled method based on vision and laser ranging is adopted. By constructing a unified optimization estimation framework, integrating visual, laser, and IMU observation data, and introducing railway track geometry rules as optimization constraints, high-precision and robust continuous dynamic estimation is achieved.
It achieves high-precision, robust, and continuous dynamic estimation of railway track conditions under complex operating environments, solving the problems of low measurement efficiency, insufficient accuracy, and error accumulation in traditional methods, and meeting the high-density operational needs of railway track inspection.
Smart Images
Figure CN122131320A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of railway track data monitoring technology, specifically to a railway track detection method and system based on machine vision and laser ranging technology. Background Technology
[0002] As an important infrastructure, railway transportation relies on the safety and smoothness of its track structure, which directly affects train safety and passenger comfort. To ensure the safety of the track structure, it is usually necessary to conduct regular and precise inspections of the track geometry parameters (such as gauge, level, elevation, and orientation) in order to achieve the maintenance and repair of railway tracks.
[0003] Currently, traditional track detection methods mainly rely on the following technologies: (1) Manual inspection and contact measurement and testing: On-site measurement is carried out using tools such as track gauges and string lines. However, in actual use on-site, this technology is often inefficient, subjective, and cannot achieve dynamic continuous measurement, thus making it difficult to meet the needs of modern railway high-density operation. (2) Using a total station / track inspection trolley for measurement and inspection: Although the measurement accuracy is high, it is a point-to-point measurement, and it also has the problems of low efficiency and complex station setup, and it also cannot capture dynamic changes in the track. (3) Track inspection vehicle measurement and detection based on inertial navigation: This is a further mainstream improvement technology. It can achieve high-speed continuous measurement by integrating inertial measurement unit, odometer and servo mechanism. However, the computing system used in this technology is generally expensive, the core sensors in the system often rely on imports, and the measurement accuracy also has the problem of drift with distance accumulation, which requires periodic absolute reference correction. (4) Pure binocular vision measurement and detection: Measurement based on visual detection technology of monocular or binocular cameras generally has the advantages of rich information and low cost. However, the distance measurement accuracy of pure vision methods, especially monocular vision, is significantly affected by factors such as camera calibration, lighting changes, and on-site texture, resulting in scale uncertainty. In actual engineering field use, it is difficult to meet the high precision and high reliability requirements of track detection. In addition, although binocular vision can measure distance, the accuracy and stability of its depth calculation in large outdoor scenes still face challenges, making it difficult to directly use as a reliable measurement benchmark.
[0004] To address this, this application proposes a railway track detection method based on vision and laser ranging. It utilizes a graph optimization joint estimation framework based on tight coupling of multiple sensors and prior geometric constraints of railway tracks to construct a unified and computable optimization model. This model deeply integrates sensor observation data with the inherent physical rules of the track, achieving high-precision, robust, and continuous dynamic calculation of track conditions, thereby solving the aforementioned technical problems. Summary of the Invention
[0005] The main objective of this invention is to provide a railway track detection method based on vision and laser ranging. By abandoning the traditional step-by-step, loosely coupled processing flow, a unified and self-consistent optimization estimation framework is constructed. By deeply fusing observation data from vision, laser, and IMU, and introducing the inherent geometric rules of railway tracks as optimization constraints, a high-precision, robust, and continuous dynamic estimation of track status is achieved under complex operating environments. This addresses the technical problems mentioned in the background art of existing track detection technologies, such as sensor error accumulation, insufficient absolute accuracy, weak scene adaptability, and lack of railway scene-specific optimization mechanisms.
[0006] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: A railway track detection method based on vision and laser ranging, comprising the following steps: S1. The sensor structure, including the binocular camera, laser rangefinder, and IMU, is fixed into an integrated measurement unit by a rigid connection mechanism. This ensures that the relative poses of each sensor in the integrated measurement unit remain constant during the measurement process. The unit is then mounted on a track inspection vehicle or mobile platform for multi-sensor calibration. Simultaneously, within the measurement area, at least one stable control point is measured and acquired using high-precision instruments such as a total station. Three-dimensional coordinates in the world coordinate system And enter it into the system database. The control point should have features that are easy to identify visually, such as a sign of a specific shape or a fixed mark on the sleeper. S2. Real-time data synchronous acquisition is performed using sensors with integrated measurement units; S3. Based on collected data and stable control points Perform instantaneous absolute pose calculation for the camera; S4. Construct the camera coordinate system coordinates of the point to be measured, and combine the instantaneous absolute pose of the camera after the solution to calculate and obtain the absolute three-dimensional coordinates of the track point to be measured; S5. To address the issues of error accumulation and insufficient utilization of track geometric constraints in traditional methods, a sliding window is constructed, and the sensor pose, absolute three-dimensional coordinates of track points, and track geometric parameters are jointly estimated within the sliding window. The optimal state estimate is finally calculated using a tightly coupled optimization method based on factor graphs. S6. Perform the final track geometry parameter extraction and result output, and finally repeat steps S2 to S5 to achieve dense measurement and parameter calculation of continuous track sections. Finally, store the coordinates and geometric parameters of all track points, and generate track smoothness reports, over-limit alarms and other information based on this. Output the final results to the display interface or upload them to the data center to support railway maintenance decisions.
[0007] Preferably, the specific calibration and correction performed in step S1 for multi-sensor calibration includes: Binocular camera intrinsic parameter calibration: Zhang Zhengyou calibration method is adopted. By shooting a checkerboard calibration board in different poses, the intrinsic parameter matrix and distortion coefficient of the left and right cameras are obtained. Binocular camera extrinsic parameter calibration: The rotation matrix and translation vector of the right camera relative to the left camera are obtained through stereo calibration, and the epipolar correction of the binocular system is finally completed. Laser rangefinder and camera extrinsic parameter calibration: Control the laser rangefinder to aim at a specific corner point on the calibration plate, and solve the direction vector and exit point position of the laser rangefinder beam in the camera coordinate system by minimizing the reprojection error or geometric constraints; IMU and camera extrinsic parameter calibration: By performing various specified movements through a handheld integrated unit, the rotation and translation relationship between the IMU coordinate system and the camera coordinate system is solved based on the hand-eye calibration algorithm. Preferably, the specific acquisition process for real-time data synchronization in S2 includes: The system controller sends a hardware synchronization signal, which simultaneously triggers the binocular camera to expose, latches the current raw IMU data (angular velocity, acceleration), and starts the laser rangefinder to measure distance; At this time, the following data is collected simultaneously: Visual data: A single frame containing known control points is simultaneously acquired using a binocular camera. Left and right views of the track points to be measured (such as the edge of the rail head and key points of the rail web); Laser distance: The laser rangefinder's spot is controlled by a set of servo pan-tilt units to precisely aim at the control points sequentially. and the point to be measured Record the corresponding precise distance values respectively. and This can be expanded to perform continuous distance measurement on multiple points to be measured; Attitude data: The IMU outputs the raw angular velocity and acceleration data at the current moment in real time.
[0008] Preferably, in step S3, the visual-laser observation of a single control point, combined with the initial attitude provided by the IMU, is used to calculate the coarse absolute pose of the camera at the current moment, which serves as the initial value for subsequent sliding window optimization. The specific operation process is as follows: S31. Perform feature extraction and recognition on the image from the left camera (main camera) to obtain known control points. Pixel coordinates in the left image ; S32. Calculate camera pose using spatial resection method: Establish the world coordinates of the control points based on the camera imaging model. Its image point coordinates The geometric relationship will determine the distance to the control point provided by the laser rangefinder. As a strong constraint, specifically, this distance value defines the precise spatial distance from the camera's optical center to the control point. Then, combined with the initial attitude angle provided by the IMU as the initial value for iterative optimization, a joint optimization objective function incorporating reprojection error and laser ranging error is constructed:
[0009] in, This represents the camera projection function, used to project 3D points through the intrinsic parameter matrix of a stereo camera. and position Projected onto the image plane, The weighting coefficients for the laser ranging error term are: and Let be the camera rotation matrix and translation vector to be solved, respectively. Furthermore, a nonlinear least squares algorithm (such as the Levenberg-Marquardt algorithm) is used to solve for the optimal rotation matrix R and translation vector T of the left camera in the world coordinate system at the current moment, which is used as the instantaneous absolute pose of the camera.
[0010] Preferably, the specific operation process of S4 includes: S41. Image Extraction and Stereo Matching of Test Points: Identify and extract the track test points from the left camera image. pixel coordinates In the right camera image, a stereo matching algorithm is used to find the corresponding image point in the left image that corresponds to the point to be measured, and its pixel coordinates are obtained. ; S42. Calculation of the coordinates of the test point based on laser distance fusion: Utilizing the principle of binocular stereo vision, according to... and Calculate the preliminary three-dimensional coordinates of the point to be measured in the left camera coordinate system. Then, the precise distance provided by the laser rangefinder is used. For the depth value in the initial three-dimensional coordinates After correction and fusion, more accurate camera coordinate system coordinates are obtained. ; The specific steps include: S421. Utilizing the principle of binocular stereoscopic vision, according to and Calculate the preliminary three-dimensional coordinates of the point to be measured in the left camera coordinate system. ,in:
[0011] in, The baseline length of the binocular camera. For camera focal length, The pixel coordinates of the camera's optical center in the image. For parallax; S422. Utilizing the precise distance provided by a laser rangefinder For the depth value in the initial three-dimensional coordinates Perform correction to obtain the corrected depth value. :
[0012] S423. Based on the corrected depth value Recalculate the 3D coordinates in the camera coordinate system :
[0013] This completes the initial correction and fusion of the three-dimensional coordinates, resulting in accurate camera coordinate system coordinates.
[0014] S43. World Coordinate Transformation: Using the instantaneous absolute pose of the camera (rotation matrix R and translation vector T) obtained in S32, the coordinates of the test point in the camera coordinate system obtained in S42 are transformed. By transforming to the world coordinate system, the absolute three-dimensional coordinates of the point to be measured are finally obtained. The conversion formula is:
[0015] in, Let the rotation matrix be the instantaneous absolute pose of the camera. It is the translation vector in the instantaneous absolute pose of the camera.
[0016] Preferably, the sliding window in S5 defines the nearest... The set of state variables at each moment is:
[0017] in, Indicates the first The pose of the integrated measurement unit in the world coordinate system at each moment. and These are the rotation matrix R and the translation vector T, respectively. This is a three-dimensional special Euclidean group used to describe the pose of objects in three-dimensional space. Used to indicate the corresponding number At that moment, the IMU's gyroscope reading was zero bias. Zero bias of accelerometer , Used to indicate the first The three-dimensional coordinates of the points to be measured on the track in the world coordinate system. Used to represent a set of track geometry parameters, including local gauge, curve superelevation, curvature, etc.
[0018] Preferably, the specific operation process of S5 is implemented by constructing a multi-source factor graph, including: A factor graph model containing the following factors is constructed, and the state variables are connected through the measured values. Finally, the joint optimization solution is performed based on the relationship of the factor graph model, and the final output includes the optimal state estimate including sensor pose, orbit point coordinates and implicit geometric parameters. The factor graph model includes: (1) Visual reprojection factor Used to constrain pose with point coordinates The error is defined as the difference between the observed value and the projected value calculated based on the camera projection model, and the error definition formula is as follows:
[0019] in, and pose respectively The corresponding rotation matrix and translation vector satisfy the following conditions: , For the first The three-dimensional coordinate vectors of the points to be measured on the track in the world coordinate system. This represents the camera projection function, used to project 3D points in the camera coordinate system. Through camera intrinsic parameter matrix Projected onto the image pixel plane, its calculation formula is:
[0020] The intrinsic parameter matrix of the stereo camera. and The first The orbital point observed at each moment Pixel coordinate observations in the left and right views; (2) IMU pre-integration factor Used to utilize IMU in adjacent time intervals and The relative motion constraints are obtained by measuring the angular velocity and acceleration between the two points using pre-integration techniques. Ultimately used to connect adjacent bits , Speed and IMU zero bias , ; (3) Laser ranging factor Used to directly constrain sensor pose with point coordinates Its error is the difference between the laser-measured distance and the geometrically calculated distance, and the error is defined by the following formula:
[0021] in, Let these be the coordinates of the point in the camera coordinate system. For laser rangefinder at time Aiming point At that time, the precise distance observations provided; (4) Absolute position factor of control points Used in known world coordinates When the control points are observed, they form a strong constraint on the system, which facilitates the direct anchoring of the system's absolute position and scale in the world coordinate system. (5) Track gauge constraint factor This is a track geometry constraint factor used to constrain the distance between a pair of left and right railhead points on the same cross section to be close to the standard gauge. (e.g., 1435mm), where the constraint error is defined as:
[0022] in, and These are the coordinates of the left and right railhead points, respectively. (6) Orbital continuity constraint factor It belongs to the category of orbital geometric constraint factors, used to constrain adjacent orbital points by utilizing the prior knowledge of the second-order continuity of the orbital space curve. The smoothness of the resulting local curves can be modeled as a smoothing regularization term for position or curvature.
[0023] (7) Curve segment superelevation constraint factor It belongs to the track geometric constraint factor and is used to associate the elevation difference of the measurement point with the design parameters to enhance the geometric consistency of the curve segment and make the elevation difference between the inner and outer rails (superelevation) of the curve segment more consistent. With curve radius and design speed There is a theoretical relationship .
[0024] Preferably, the solution method for joint optimization based on factor graph model relationships in S5 is as follows: Constructing the factors in the factor graph model as nonlinear least squares formulas, we have:
[0025] in, The L2 norm (Euclidean norm) of a vector is represented by its square, which is the sum of the squares of the error components. The covariance weighted norm (Maharanobis distance) is defined as follows: Used to determine the uncertainty of the sensor (derived from the covariance matrix). The error is reasonably weighted (characterized). For Huber kernel function, The covariance matrix of each sensor, For geometric constraint weights, For the set of all geometric constraint errors, The constraint error for the visual reprojection factor. The constraint error for the IMU pre-integration factor. The constraint error of the laser ranging factor, The constraint error is the absolute position factor of the control point; Finally, the Levenberg-Marquardt algorithm is used to solve the nonlinear least squares formula within a sliding window to obtain the optimal state estimate. .
[0026] Preferably, the specific output in S6 is orbital geometric parameters, which are based on the orbital point cloud obtained from the optimal state estimation. Direct calculation yields: Track gauge calculation: Select the left and right rail head points of the same cross section, calculate the three-dimensional Euclidean distance, and use it as the track gauge parameter; Elevation calculation: Select a sequence of center points of the rail top along the longitudinal direction of the track, calculate the elevation difference between adjacent points or fit the longitudinal profile curve as its elevation parameter; Direction calculation: Project the point cloud onto the horizontal plane, fit the center line of the track and calculate the lateral offset as its direction parameter; Horizontal calculation: Calculate the elevation difference between the left and right rail head points of the same cross section, and use it as its horizontal parameter.
[0027] On the other hand, the present invention also discloses a railway track inspection system based on vision and laser ranging, including an integrated measurement unit and a data processing unit located on a track inspection vehicle or mobile platform: The integrated measurement unit includes sensors such as a binocular camera, a laser rangefinder, and an IMU, which are used to collect specified data of the railway track and send it to the data processing unit. The data processing unit includes a receiver, a memory, and a processor. The receiver is used to receive data transmitted by the data processing unit. The memory stores a computer program. When the receiver receives the data and the computer program is executed by the processor, the processor performs calculation steps as described in any of the above methods.
[0028] As can be seen from the above technical solution, the present invention provides a railway track detection method based on vision and laser ranging. Compared with the prior art, the present invention has the following advantages: 1. This invention abandons the traditional step-by-step, loosely coupled processing flow and constructs a unified and self-consistent optimization estimation framework. By deeply integrating observation data from vision, laser, and IMU, and by uniquely introducing the inherent geometric rules of railway tracks as optimization constraints, it achieves high-precision, robust, and continuous dynamic estimation of track status under complex operating environments.
[0029] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description
[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall operation process of the present invention. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] For details in the embodiments, please refer to Figure 1 .
[0033] like Figure 1 As shown in the figure, the railway track detection method based on vision and laser ranging proposed in this embodiment of the invention includes the following steps: S1: System initialization and calibration phase.
[0034] S1.1: Hardware system integration.
[0035] The binocular camera, laser rangefinder, and IMU are fixed together as an integrated measurement unit through a rigid connection mechanism to ensure that the relative pose of each sensor remains constant during the measurement process, and are installed on a track inspection vehicle or mobile platform.
[0036] S1.2: Multi-sensor calibration.
[0037] S1.2.1 Binocular camera intrinsic parameter calibration: Zhang Zhengyou calibration method was adopted. The intrinsic parameter matrix and distortion coefficient of the left and right cameras were obtained by shooting the checkerboard calibration board in different poses.
[0038] S1.2.2 Binocular Camera Extrinsic Parameter Calibration: Through stereo calibration, the rotation matrix and translation vector of the right camera relative to the left camera are obtained to complete the epipolar correction of the binocular system.
[0039] S1.2.3 Calibration of laser rangefinder and camera extrinsic parameters: Control the laser rangefinder to aim at a specific corner point on the calibration plate, and solve for the direction vector and exit point position of the laser rangefinder beam in the camera coordinate system by minimizing the reprojection error or geometric constraints.
[0040] S1.2.4 IMU and Camera Extrinsic Parameter Calibration: Through various movements performed by the handheld integrated unit, the rotation and translation relationship between the IMU coordinate system and the camera coordinate system is solved based on the hand-eye calibration algorithm.
[0041] S1.3: Control point information entry.
[0042] Within the measurement area, at least one stable control point should be measured using high-precision instruments such as a total station. Three-dimensional coordinates in the world coordinate system This information is then entered into the system database. The control point should have easily visually identifiable features, such as a specially shaped sign or a fixed mark on the sleeper.
[0043] S2: Real-time data synchronization and acquisition stage.
[0044] S2.1: Synchronous triggering.
[0045] The system controller sends a hardware synchronization signal, which simultaneously triggers the binocular camera to expose, latches the current raw IMU data (angular velocity, acceleration), and starts the laser rangefinder to measure distance.
[0046] S2.2: Data Acquisition.
[0047] S2.2.1 Visual Data Acquisition: Binocular cameras simultaneously acquire one frame containing known control points. Left and right views of the track points to be measured (such as the edge of the rail head and key points of the rail waist).
[0048] S2.2.2 Laser Distance Acquisition: The laser rangefinder's spot is controlled by a servo pan-tilt unit to precisely aim at the control points sequentially. and the point to be measured Record the corresponding precise distance values respectively. and It can be expanded to perform continuous distance measurement on multiple points to be measured.
[0049] S2.2.3 Attitude Data Acquisition: The IMU outputs the raw angular velocity and acceleration data at the current moment in real time.
[0050] S3: Instantaneous absolute pose calculation of the camera based on control points.
[0051] This step utilizes visual-laser observations from a single control point, combined with the initial attitude provided by the IMU, to calculate the camera's coarse absolute pose at the current moment, which serves as the initial value for subsequent sliding window optimization.
[0052] S3.1: Control point image extraction: Feature extraction and recognition are performed on the images from the left camera (main camera) to obtain known control points. Pixel coordinates in the left image .
[0053] S3.2: Spatial rear intersection solution for camera pose: Establish the world coordinates of the control points based on the camera imaging model. Its image point coordinates The geometric relationship.
[0054] The distance to the control point provided by the laser rangefinder This serves as a strong constraint. Specifically, this distance value defines the precise spatial distance from the camera's optical center to the control point.
[0055] By combining the initial attitude angle provided by the IMU as the initial value for iterative optimization, a joint optimization objective function that includes reprojection error and laser ranging error is constructed.
[0056] A nonlinear least squares algorithm (such as the Levenberg-Marquardt algorithm) is used to solve for the optimal rotation matrix R and translation vector T of the left camera in the world coordinate system at the current moment, which is the instantaneous absolute pose of the camera.
[0057] S4: Calculation of the absolute three-dimensional coordinates of the point to be measured.
[0058] S4.1: Image extraction and stereo matching of the points to be measured: Identify and extract the track measurement points from the left camera image. pixel coordinates .
[0059] In the right camera image, a stereo matching algorithm is used to find the corresponding image point of the same name as the point to be measured in the left image, and its pixel coordinates are obtained. .
[0060] S4.2: Calculation of the coordinates of the point to be measured based on the fused laser distance: Utilizing the principle of binocular stereo vision, based on and Calculate the preliminary three-dimensional coordinates of the point to be measured in the left camera coordinate system. .
[0061] Using the precise distance provided by a laser rangefinder For the depth value in the initial three-dimensional coordinates Correction and fusion are performed to obtain a more accurate camera coordinate system. .
[0062] S4.3: World coordinate transformation: Using the camera pose (rotation matrix R and translation vector T) obtained in step S3.2, the coordinates of the camera coordinate system of the point to be measured obtained in S4.2 are... By converting to the world coordinate system, the absolute three-dimensional coordinates of the point to be measured are finally obtained. The conversion formula is: .
[0063] S5: Multi-sensor data fusion and joint estimation of orbital state based on graph optimization.
[0064] To address the issues of error accumulation and underutilization of track geometric constraints in traditional methods, this step proposes a tightly coupled optimization method based on factor graphs, which jointly estimates sensor pose, three-dimensional coordinates of track points, and track geometric parameters within a sliding window.
[0065] S5.1: Definition of Sliding Window State Variables Define a sliding window (most recent) The set of state variables within a given time period is:
[0066] in: Indicates the first The pose of the integrated measurement unit in the world coordinate system at each moment; This indicates that the gyroscope and accelerometer of the IMU have zero bias at the corresponding moment; Indicates the first The three-dimensional coordinates of the points to be measured on the track in the world coordinate system; It represents the set of track geometry parameters, including local gauge, curve superelevation, curvature, etc.
[0067] S5.2: Construction of multi-source factor graphs.
[0068] Construct a factor graph model containing the following factors, connecting the state variables through measurements: Visual reprojection factor For time Observed orbital points Its pixel observation values in the left and right views of the binocular camera are This factor constrains the pose. with point coordinates The error is defined as the difference between the observed value and the projected value calculated based on the camera projection model:
[0069] in, and pose respectively The corresponding rotation matrix and translation vector satisfy the following conditions: , For the first The three-dimensional coordinate vectors of the points to be measured on the track in the world coordinate system. This represents the camera projection function, used to project 3D points in the camera coordinate system. Through camera intrinsic parameter matrix Projected onto the image pixel plane, its calculation formula is: , The intrinsic parameter matrix of the stereo camera. and The first The orbital point observed at each moment Pixel coordinate observations in the left and right views.
[0070] IMU pre-integration factor : Utilizing IMU in adjacent time intervals and The angular velocity and acceleration measurements between the two points are used to obtain a relative motion constraint through pre-integration techniques. This factor connects adjacent bits. Speed and IMU zero bias .
[0071] Laser ranging factor When the laser rangefinder is at time Aiming point At that time, it provides accurate distance observations. This factor directly constrains the sensor pose. with point coordinates Its error is the difference between the laser-measured distance and the geometrically calculated distance:
[0072] in Let be the coordinates of the point in the camera coordinate system.
[0073] absolute position factor of control points For known world coordinates The control points, when observed, form a strong constraint that directly anchors the absolute position and scale of the system in the world coordinate system.
[0074] Orbital geometric constraint factor: Track gauge constraint factor For a pair of left and right rail head points on the same cross section and Its distance should be close to the standard gauge. (e.g., 1435mm). The error is defined as:
[0075] Orbit continuity constraint factor Using the prior knowledge of the second-order continuity of the orbital space curve, adjacent orbital points are constrained. The smoothness of the resulting local curves can be modeled as a smoothing regularization term for position or curvature.
[0076] Super-high constraint factor of curve segment In curved sections, there is a height difference between the inner and outer rails (superelevation). With curve radius and design speed There is a theoretical relationship This factor correlates the elevation difference of the measurement points with the design parameters, enhancing the geometric consistency of the curve segment.
[0077] S5.3: Joint optimization solution.
[0078] The above factors are constructed as a nonlinear least squares problem:
[0079] in For Huber kernel function, The covariance matrix of each sensor, For geometric constraint weights, This is the set of all geometric constraint errors.
[0080] The Levenberg-Marquardt algorithm is used to solve the problem within a sliding window, and the optimal state estimate is output. This includes sensor pose, orbital point coordinates, and implicit geometric parameters.
[0081] S6: Track geometry parameter extraction and result output.
[0082] S6.1: Direct extraction of orbital geometry parameters.
[0083] The orbital point cloud obtained based on S5.3 optimization Directly calculate the orbital geometry parameters: Track gauge: Select the left and right rail head points of the same cross section and calculate the three-dimensional Euclidean distance.
[0084] Elevation: Select a sequence of center points of the top of the track along the longitudinal direction and calculate the elevation difference between adjacent points or fit the longitudinal profile curve.
[0085] Direction: Project the point cloud onto the horizontal plane, fit the orbit centerline, and calculate the lateral offset.
[0086] Horizontal: Calculate the elevation difference between the left and right rail head points on the same cross section.
[0087] S6.2: Result Output and Data Processing Steps S2 through S5 are executed iteratively to perform intensive measurements and parameter calculations on continuous track sections. All track point coordinates and geometric parameters are stored, and track smoothness reports, over-limit alarms, and other information are generated based on this data. The final results are output to a display interface or uploaded to a data center to support railway maintenance decision-making.
[0088] The above steps, by abandoning the traditional step-by-step and loosely coupled processing flow, construct a unified and self-consistent optimization estimation framework. By deeply integrating observation data from vision, laser, and IMU, and by uniquely introducing the inherent geometric rules of railway tracks as optimization constraints, high-precision, robust, and continuous dynamic estimation of track status is achieved under complex operating environments.
[0089] On the other hand, the present invention also discloses a railway track inspection system based on vision and laser ranging, including an integrated measurement unit and a data processing unit located on a track inspection vehicle or mobile platform: The integrated measurement unit includes sensors such as a binocular camera, a laser rangefinder, and an IMU, which are used to collect specified data of the railway track and send it to the data processing unit. The data processing unit includes a receiver, a memory, and a processor. The receiver is used to receive data transmitted by the data processing unit. The memory stores a computer program. When the receiver receives the data and the computer program is executed by the processor, the processor performs the calculation steps as described in the above embodiments.
[0090] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the data calculation process in any of the vision-based and laser ranging-based railway track detection methods in the above embodiments.
[0091] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.
[0092] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the data calculation process in the above-mentioned railway track detection method based on vision and laser ranging.
[0093] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.
[0094] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0095] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0096] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0097] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.
[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0099] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.
[0100] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
Claims
1. A railway track detection method based on vision and laser ranging, characterized in that, include: S1. Sensors including a binocular camera, laser rangefinder, and IMU are fixed as an integrated measurement unit. The relative pose of the sensors in the integrated measurement unit is constant, and it is installed on a track inspection vehicle or mobile platform for multi-sensor calibration. At the same time, within the measurement area, at least one stable control point is measured and acquired. Three-dimensional coordinates in the world coordinate system; S2. Real-time data synchronous acquisition is performed using sensors with integrated measurement units; S3. Based on collected data and stable control points Perform instantaneous absolute pose calculation for the camera; S4. Construct the camera coordinate system coordinates of the point to be measured, and combine the instantaneous absolute pose of the camera after the solution to calculate and obtain the absolute three-dimensional coordinates of the track point to be measured; S5. Construct a sliding window and jointly estimate the sensor pose, the absolute three-dimensional coordinates of the orbit points, and the orbital geometric parameters within the sliding window to finally obtain the optimal state estimate.
2. The railway track detection method based on vision and laser ranging as described in claim 1, characterized in that, The specific calibration and correction performed in S1 for multiple sensors includes: Binocular camera intrinsic parameter calibration: Zhang Zhengyou calibration method is adopted. By shooting a checkerboard calibration board in different poses, the intrinsic parameter matrix and distortion coefficient of the left and right cameras are obtained. Binocular camera extrinsic calibration: Obtain the rotation matrix and translation vector of the right camera relative to the left camera using the stereo calibration method; Laser rangefinder and camera extrinsic parameter calibration: Control the laser rangefinder to aim at a specific corner point on the calibration plate, and solve the direction vector and exit point position of the laser rangefinder beam in the camera coordinate system by minimizing the reprojection error or geometric constraints; IMU and camera extrinsic parameter calibration: By performing a specified motion through a handheld integrated unit, the rotation and translation relationship between the IMU coordinate system and the camera coordinate system is solved based on a hand-eye calibration algorithm.
3. The railway track detection method based on vision and laser ranging as described in claim 1, characterized in that, The specific data acquisition process for real-time data synchronization in S2 includes: The system controller sends a hardware synchronization signal, which simultaneously triggers the binocular camera to expose, latches the current IMU raw data, and starts the laser rangefinder to measure distance. At this time, the following data is collected simultaneously: Visual data: A single frame containing known control points is simultaneously acquired using a binocular camera. Left and right views of the track's test points; Laser distance: The laser rangefinder's spot is controlled by a set of servo pan-tilt units to precisely aim at the control points sequentially. and the point to be measured Record the corresponding precise distance values respectively. and ; Attitude data: The IMU outputs the raw angular velocity and acceleration data at the current moment in real time.
4. The railway track detection method based on vision and laser ranging as described in claim 1, characterized in that, The specific operation process in S3 is as follows: S31. Perform feature extraction and recognition on the left camera image to obtain known control points. Pixel coordinates in the left image ; S32. Establish the world coordinates of the control points based on the camera imaging model. Its image point coordinates The geometric relationship will determine the distance to the control point provided by the laser rangefinder. As a strong constraint, and using the initial attitude angle provided by the IMU as the initial value for iterative optimization, a joint optimization objective function including reprojection error and laser ranging error is constructed as follows: in, This represents the camera projection function, used to project 3D points through the intrinsic parameter matrix of a stereo camera. and position Projected onto the image plane, The weighting coefficients for the laser ranging error term are: and Let be the camera rotation matrix and translation vector to be solved, respectively. Then, a nonlinear least squares algorithm is used to solve for the optimal rotation matrix R and translation vector T of the left camera in the world coordinate system at the current moment, which is used as the instantaneous absolute pose of the camera.
5. The railway track detection method based on vision and laser ranging as described in claim 4, characterized in that, The specific operation process of S4 includes: S41. Identify and extract the track measurement points in the left camera image. pixel coordinates In the right camera image, a stereo matching algorithm is used to find the corresponding image point in the left image that corresponds to the point to be measured, and its pixel coordinates are obtained. ; S42. Utilizing the principle of binocular stereoscopic vision, according to and Calculate the preliminary three-dimensional coordinates of the point to be measured in the left camera coordinate system. Then, the precise distance provided by the laser rangefinder is used. For the depth value in the initial three-dimensional coordinates After correction and fusion, the camera coordinate system coordinates are finally obtained. ; S43. Using the instantaneous absolute pose of the camera, the coordinates of the point to be measured in the camera coordinate system are... By transforming to the world coordinate system, the absolute three-dimensional coordinates of the point to be measured are finally obtained. The conversion formula is: in, Let be the rotation matrix in the instantaneous absolute pose of the camera. It is the translation vector in the instantaneous absolute pose of the camera.
6. The railway track detection method based on vision and laser ranging as described in claim 1, characterized in that, The sliding window in S5 defines the nearest... The set of state variables at each moment is: in, Indicates the first The pose of the integrated measurement unit in the world coordinate system at each moment. and These are the rotation matrix R and the translation vector T, respectively. This is a three-dimensional special Euclidean group used to describe the pose of objects in three-dimensional space. Used to indicate the corresponding number At that moment, the IMU's gyroscope reading was zero bias. Zero bias of accelerometer , Used to indicate the first The three-dimensional coordinates of the points to be measured on the track in the world coordinate system. Used to represent a set of orbital geometric parameters.
7. The railway track detection method based on vision and laser ranging as described in claim 6, characterized in that, The specific operation process of S5 includes: A factor graph model containing the following factors is constructed, and each state variable is connected through the measured values. Finally, the joint optimization solution is performed based on the relationship of the factor graph model, and the final output includes the optimal state estimate including sensor pose, orbit point coordinates and implicit geometric parameters. The factor graph model includes: (1) Visual reprojection factor Used to constrain pose with point coordinates The error is defined as the difference between the observed value and the projected value calculated based on the camera projection model, and the error definition formula is as follows: in, and pose respectively The corresponding rotation matrix and translation vector satisfy the following conditions: , For the first The three-dimensional coordinate vectors of the points to be measured on the track in the world coordinate system. This represents the camera projection function, used to project 3D points in the camera coordinate system. Through camera intrinsic parameter matrix Projected onto the image pixel plane, its calculation formula is: The intrinsic parameter matrix of the stereo camera. and The first The orbital point observed at each moment Pixel coordinate observations in the left and right views; (2) IMU pre-integration factor Used to utilize IMU in adjacent time intervals and The relative motion constraints are obtained by measuring the angular velocity and acceleration between the two points using pre-integration techniques. Ultimately used to connect adjacent bits , Speed and IMU zero bias , ; (3) Laser ranging factor Used to directly constrain sensor pose with point coordinates Its error is the difference between the laser-measured distance and the geometrically calculated distance, and the error is defined by the following formula: in, Let these be the coordinates of the point in the camera coordinate system. For laser rangefinder at time Aiming point At that time, the precise distance observations provided; (4) Absolute position factor of control points Used in known world coordinates When the control points are observed, they form a strong constraint on them; (5) Track gauge constraint factor Constrain a pair of left and right rail head points on the same cross section so that the distance between them is close to the standard gauge. The constraint error is defined as follows: in, and These are the coordinates of the left and right railhead points, respectively. (6) Orbital continuity constraint factor By utilizing the prior knowledge of the second-order continuity of the orbital space curve, adjacent orbital points are constrained. The smoothness of the local curves formed. (7) Curve segment superelevation constraint factor It is used to correlate the elevation difference of measurement points with design parameters, so that the elevation difference between the inner and outer rails of the curve segment is... With curve radius and design speed Existence Relationship .
8. The railway track detection method based on vision and laser ranging as described in claim 7, characterized in that, The solution method for joint optimization based on factor graph model relationships in S5 is as follows: Constructing the factors in the factor graph model as nonlinear least squares formulas, we have: in, The L2 norm of a vector. This represents the covariance weighted norm, used to reasonably weight errors based on sensor uncertainties. For Huber kernel function; The covariance matrix of each sensor; These are geometric constraint weights; For the set of all geometric constraint errors; The constraint error for the visual reprojection factor; The constraint error for the IMU pre-integration factor; This represents the constraint error of the laser ranging factor; The constraint error is the absolute position factor of the control point; Finally, the Levenberg-Marquardt algorithm is used to solve the nonlinear least squares formula within a sliding window to obtain the optimal state estimate. .
9. The railway track detection method based on vision and laser ranging as described in claim 1, characterized in that, It also includes step S6, which is used to extract and output the final orbital geometric parameters, and finally iterates through steps S2 to S5 to achieve dense measurement and parameter calculation of continuous orbital sections. Specifically, the output is the orbital geometric parameters, which are based on the orbital point cloud obtained from the optimal state estimation. Direct calculation yields: Track gauge calculation: Select the left and right rail head points of the same cross section, calculate the three-dimensional Euclidean distance, and use it as the track gauge parameter; Elevation calculation: Select a sequence of center points of the rail top along the longitudinal direction of the track, calculate the elevation difference between adjacent points or fit the longitudinal profile curve as its elevation parameter; Direction calculation: Project the point cloud onto the horizontal plane, fit the center line of the track and calculate the lateral offset as its direction parameter; Horizontal calculation: Calculate the elevation difference between the left and right rail head points of the same cross section, and use it as its horizontal parameter.
10. A railway track inspection system based on vision and laser ranging, characterized in that, This includes an integrated measurement unit and a data processing unit located on a track inspection vehicle or mobile platform. The integrated measurement unit includes sensors such as a binocular camera, a laser rangefinder, and an IMU, which are used to collect specified data of the railway track and send it to the data processing unit. A data processing unit is provided with a receiver, a memory, and a processor. The receiver is used to receive data transmitted by the data processing unit. The memory stores a computer program. When the receiver receives the data and the computer program is executed by the processor, the processor causes the processor to perform the calculation steps in the method as described in any one of claims 1 to 9.