A method for railway defect localization and automatic navigation based on multimodal fusion positioning
By constructing a multimodal fusion positioning and ground feature association database, and combining BeiDou coordinates and sleeper numbers, collaborative navigation and mileage calibration of railway defect inspection equipment are achieved. This solves the problems of insufficient positioning accuracy and cumulative error in existing technologies, and improves the accuracy and automation level of defect positioning.
Patent Information
- Application Number
- CN202610939412.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-26
- Publication Date
- 2026-11-10
AI Technical Summary
The existing railway defect inspection and positioning accuracy is insufficient, the single navigation method is constrained by the environment, and the cumulative mileage error is difficult to correct, resulting in the equipment being unable to accurately stop at the defect location.
By constructing a multimodal fusion positioning and ground feature association database, and combining BeiDou coordinates, operating mileage values and sleeper numbers, image data is collected in real time for multi-source information fusion, enabling collaborative navigation and mileage calibration of navigation equipment, and finally relying on sleeper numbers to locate the defect.
It improves the accuracy and automation level of railway defect location, reduces the labor cost of line maintenance, and ensures that equipment can navigate stably and stop accurately at defect locations in complex environments.
Smart Images

Figure CN123066818A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent track inspection technology, and more specifically, relates to a method and system for locating railway defects and automatic navigation based on multimodal fusion positioning. Background Technology
[0002] my country's railway network has been expanding year by year, and rail transit has become a core component of the domestic transportation system. Rail lines are subjected to multiple external influences over long periods, including train reciprocating loads, rain and snow erosion, and geological subsidence. Various line defects, such as rail damage, ballast bed damage, and roadbed deformation, occur routinely. Locating these defects is a crucial step in line maintenance and ensuring railway safety. For a long time, railway defect inspections have relied heavily on on-site personnel conducting foot surveys and manually recording the mileage and geographical location of defects. Manual inspections are not only time-consuming and costly, but also extremely difficult in complex environments such as mountainous tunnels and remote open fields. Defect locations identified visually are often inaccurate, frequently leading to difficulties in locating them during subsequent maintenance work.
[0003] While automated inspection equipment has been gradually deployed, mainstream positioning methods have significant shortcomings. Most devices rely solely on a single data source—either BeiDou satellite navigation or wheel odometers—for positioning. However, BeiDou signals are easily interrupted in tunnels, dense forests, and elevated roadways. When relying solely on odometers to calculate position, long-term wheel wear accumulates mileage errors. Neither of these positioning methods can achieve stable, high-precision navigation across the entire road segment. A few inspection devices equipped with visual recognition only use a single reference point for position correction, failing to establish data correlations between ground features, sleepers, and defect locations.
[0004] Currently, the industry lacks a unified multimodal benchmark database. Landmark features and sleeper physical numbers along the track cannot be linked to the mileage and satellite coordinates of the defects for storage. Inspection equipment can only roughly approach the defect area during long-distance travel, making it difficult to uniformly correct the mileage benchmark when nearing the target location. The uneven spacing of sleepers in turnout sections, coupled with the lack of targeted compensation logic in existing technology, further exacerbates the point offset due to errors in sleeper number calculation, preventing equipment from accurately stopping at the original defect location.
[0005] The industry urgently needs a positioning and navigation solution that integrates multi-source information, relies on multi-dimensional data linkage to eliminate various system errors, overcome the problem of inaccurate positioning in complex sections, promote the transformation of railway defect inspection from manual and extensive operations to automated and accurate operations, reduce maintenance costs, and improve the reliability and efficiency of railway line maintenance from the source. Summary of the Invention
[0006] This invention aims to overcome the shortcomings of existing railway defect inspection and positioning, such as insufficient accuracy, environmental constraints of single navigation methods, and difficulty in correcting cumulative mileage errors. It relies on the multimodal fusion of Beidou, odometer, and visual recognition to build a line-related database, complete navigation correction and mileage calibration in stages, and achieve fixed-point stopping of defects by sleeper numbering, thereby improving the automation level of railway inspection and the accuracy of defect positioning, and reducing the manpower cost of line maintenance.
[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, as a first aspect of this invention, the present invention provides a railway defect location and automatic navigation method based on multimodal fusion positioning, comprising: S1. Acquire defect data and salient feature data along the railway line, and construct a multimodal positioning and feature association database; the database includes defect feature vectors and salient feature feature vectors; both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values and sleeper physical numbers; the salient feature feature vectors also contain feature types and feature image data; S2. Control the navigation device to move towards the target defect location. During the movement, perform coordinated navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, collect real-time images of the landscape ahead and the sleepers below, identify the prominent features and sleepers currently being passed, and compare them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. S3. When the navigation device moves to a prominent feature adjacent to the target defect location, the current mileage output value of the navigation device is obtained; based on the current mileage output value and the standard operating mileage value of the prominent feature in the database, the mileage cumulative error is calculated; the mileage cumulative error is used to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. S4. After completing the internal mileage coordinate system compensation calibration, increase the weight of sleeper identification, and calculate the physical number of the sleeper currently being crossed based on the calibrated internal mileage coordinate system and the real-time identified sleepers; when the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, control the navigation device to stop, and complete the locking of the target defect location.
[0008] Furthermore, in S1, the specific process of constructing the multimodal positioning and ground feature association database includes: The laser image detection system is used to automatically detect railway lines on-site, and to collect three-dimensional lidar data and high-definition image data of the railway lines. Based on the three-dimensional lidar data and the high-definition image data, the defects are identified and extracted, and three types of positioning data are obtained for each defect: Beidou location, sleeper number, and operating mileage. Based on the three-dimensional lidar data and the high-definition image data, the system automatically extracts data on significant scene features on the detection line. The data on significant scene features includes signal lights, catenary poles, bridge and tunnel locations, and curve features. The system also obtains three types of positioning data for each significant scene feature: BeiDou location, sleeper number, and operating mileage. A unified timestamp is assigned to each disease data and each significant scene feature data, and the three-dimensional lidar data, high-definition image data and positioning data are spatiotemporally aligned based on the timestamp; The BeiDou location, sleeper number, and operating mileage of each aligned defect are encapsulated to generate the defect feature vector. The BeiDou location, sleeper number, operating mileage, feature type, and feature image data of each aligned salient scene feature are encapsulated to generate the salient feature feature vector. This ensures that each detected defect corresponds to three types of positioning data: BeiDou location, sleeper number, and operating mileage, as well as associated salient scene feature data of the line.
[0009] Furthermore, in step S2, collaborative navigation based on the BeiDou coordinates and the operational mileage value is performed to approximate the target disease location, specifically including: Real-time assessment of BeiDou signal quality indicators; When the quality index is higher than the preset threshold, the real-time position of the navigation device is calculated based on BeiDou positioning data as the primary source and odometer data as the secondary source. When the quality index is lower than or equal to the preset threshold, the BeiDou signal is determined to be restricted, and the system automatically downgrades to a navigation mode based on odometer data. The system then uses the heading angle and acceleration data output by the inertial measurement unit to perform integral compensation on the odometer data to maintain continuous navigation in tunnel or bridge sections.
[0010] Furthermore, in step S2, real-time acquisition of images of the foreground landscape and the sleepers below, and identification of prominent features and sleepers currently being passed, specifically includes: The system acquires real-time images of the foreground landscape and extracts the visual features of current ground features; obtains the current estimated position of the navigation device, and retrieves candidate salient features from the multimodal positioning and ground feature association database with the estimated position as the center and a preset search radius; performs similarity matching between the visual features of the current ground features and the ground feature image data of the candidate salient features; when the matching similarity exceeds a preset threshold, the identification is deemed successful, and the current estimated mileage of the navigation device is corrected to the first level using the standard operating mileage value of the identified salient features in the database; Bottom sleeper images are acquired at a trigger frequency that adapts to the speed of the navigation device, and time-series verification is performed in conjunction with the pulse signals output by the wheel encoders to eliminate redundant or invalid images caused by sudden speed changes. Feature extraction is performed on the verified sleeper images to obtain the surface physical number or texture features of the current sleeper. The obtained features are matched with the sleeper physical number sequence of the corresponding section in the database using a sliding window. When the match is successful, the current BeiDou coordinates of the navigation device are corrected using the standard spatial coordinates of the sleeper in the database. Calculate the spatial deviation between the first-level corrected mileage position and the second-level corrected BeiDou coordinates; when the spatial deviation is less than a preset safety threshold, merge the correction results of the two to update the real-time position of the navigation device; when the spatial deviation is greater than or equal to the safety threshold, determine that the current identification result is abnormal, remove the correction data, and maintain the estimated position of the navigation device before correction to continue driving until the next valid object or sleeper triggers the correction.
[0011] Furthermore, in step S3, the internal mileage coordinate system of the navigation device is compensated and calibrated using the accumulated mileage error, specifically including: Obtaining navigation device information after the preceding steps Odometry output value at a prominent feature And the current number Odometry output value at a prominent feature ; Calculate the odometry interval measurement increment between two adjacent salient features. And obtain the corresponding standard interval physical distance in the multimodal positioning and ground feature association database. ; Based on the above and The ratio of the two values is used to calculate the real-time wheel diameter wear ratio coefficient of the odometer wheel; simultaneously, the formula is used... Calculate the absolute mileage drift error of the current node. ; The absolute mileage drift error As a zero-point compensation, it is directly superimposed on the current internal mileage coordinate system of the navigation device to eliminate the cumulative absolute deviation; The real-time wheel diameter wear ratio coefficient is used to update the underlying pulse equivalent of the odometer in order to correct the mileage ratio calculation error caused by wheel wear during subsequent driving. Obtain the first position covariance matrix output by the BeiDou positioning module before calibration, the second position covariance matrix calculated by the odometer, and the third position covariance matrix output by the visual ground feature positioning. When the absolute mileage drift error When the preset error threshold is exceeded, the odometer is determined to be in a high error divergence state. The weight of the odometer positioning in the Kalman filter fusion is reduced by increasing the trace value of the second position covariance matrix. Simultaneously, the sharpness index and matching confidence of the currently identified salient ground features are extracted, and the trace value of the third location covariance matrix is dynamically reduced based on the sharpness index and matching confidence to improve the fusion weight of visual ground feature localization.
[0012] Furthermore, in step S4, after completing the internal mileage coordinate system compensation calibration, the weight of sleeper identification is increased, specifically including: Obtain the residual error estimate of the internal mileage coordinate system after compensation and calibration, as well as the recognition confidence of the real-time acquired bottom sleeper images; Based on the residual error estimate, an odometer observation noise covariance matrix is constructed, and based on the identification confidence, a sleeper visual observation noise covariance matrix is constructed. The odometer observation noise covariance matrix and the sleeper visual observation noise covariance matrix are input into the extended Kalman filter as measurement update parameters; When the residual error estimate is less than the preset convergence threshold and the recognition confidence is greater than the preset confidence threshold, the diagonal element values of the sleeper visual observation noise covariance matrix are reduced, and the weight coefficients corresponding to the sleeper observation components in the extended Kalman filter gain matrix are increased, so that the fusion state estimation result converges to the sleeper recognition result, thereby completing the dynamic improvement of the sleeper recognition weight.
[0013] Furthermore, in step S4, calculating the physical number of the sleeper currently being crossed specifically includes: The physical sleeper numbers corresponding to the nearby prominent features for which compensation calibration has been completed. As the benchmark starting point; Count the number of sleepers identified in real time from the nearby prominent features to the current location. ; Through formula Calculate the current physical number of the sleeper ; When the navigation device is detected to be in a turnout section, a turnout section sleeper spacing correction coefficient is introduced for the number of sleepers. Compensation will be provided to eliminate errors in numbering calculations caused by variations in sleeper type and uneven spacing at turnouts.
[0014] Furthermore, the navigation device includes a control box, a forward landscape vision module, a sleeper counting module, and a high-precision navigation module; The foreground landscape vision module includes an area array camera, used to acquire foreground landscape images and identify prominent scene features; The sleeper counting module includes a bottom camera and an encoder, which are used to acquire images of the bottom sleepers and count the sleepers by combining encoder pulses. The control box includes a synchronization module, an industrial computer, and an edge computing module. The synchronization module is used to trigger hardware-level time synchronization of the forward landscape vision module, the sleeper counting module, and the high-precision navigation module. The edge computing module is used to perform real-time inference and recognition on the acquired images and transmit the recognition results to the industrial computer for multimodal data fusion and navigation decision-making.
[0015] As a second aspect of the present invention, a railway defect location and automatic navigation system based on multimodal fusion positioning is also provided, comprising: A relational database construction unit is used to acquire railway defect data and salient feature data to construct a multimodal positioning and feature association database. The database includes defect feature vectors and salient feature feature vectors. Both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values, and sleeper physical numbers. The salient feature feature vectors also contain feature types and feature image data. The collaborative navigation and correction unit is used to control the navigation device to move towards the target defect location. During the movement, it performs collaborative navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, it collects images of the landscape ahead and the sleepers below in real time, identifies the salient features and sleepers that are currently being passed, and compares them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. The mileage compensation calibration unit is used to obtain the current mileage output value of the navigation device when the navigation device moves to a nearby prominent feature preceding the target defect location; calculate the mileage accumulation error based on the current mileage output value and the standard operating mileage value of the nearby prominent feature in the database; and use the mileage accumulation error to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. The target defect location locking unit is used to increase the weight of sleeper identification after the internal mileage coordinate system compensation calibration is completed. Based on the calibrated internal mileage coordinate system and the real-time identified sleepers, it calculates the physical number of the sleeper currently being crossed. When the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, it controls the navigation device to stop and completes the locking of the target defect location.
[0016] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor to provide a method for locating railway defects and automatic navigation based on multimodal fusion positioning as described in any one of the claims.
[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The railway defect location and automatic navigation method based on multimodal fusion positioning of this invention collects railway defect and salient feature data and constructs a related database. The defect feature vector and salient feature feature vector uniformly include BeiDou coordinates, operating mileage, and sleeper physical numbers, and additionally supplements the feature type and corresponding image data in the feature features. The unified and standardized data storage format binds the multi-dimensional geographic information of line defect locations, track landmarks, and sleeper entities, allowing the same set of reference data to be used for subsequent navigation operations such as data retrieval, image feature comparison, and mileage value verification, solving the problems of scattered and independent positioning information and mismatched reference benchmarks in the past. The standardized vector storage structure can ensure the data retrieval efficiency of inspection equipment throughout the process, avoid matching failures caused by messy information formats at the data source level, and provide data basis for position correction and mileage calibration throughout the process.
[0018] 2. The railway defect location and automatic navigation method based on multimodal fusion positioning of the present invention completes long-distance navigation operations through the collaborative use of BeiDou coordinates and operating mileage. During the equipment's journey, it continuously collects images from both the front and back ends, identifies road features and sleeper information along the route, and performs two-level position corrections by referencing a database. Simultaneously, it verifies the spatial deviation of the two-level correction results before determining the position update logic. This collaborative navigation and hierarchical correction mechanism can flexibly switch navigation modes according to the quality of BeiDou signals. In tunnels and bridge-tunnel sections where satellites are obstructed, it relies on mileage data to continue navigation tasks. Road features and sleepers are two types of reference objects that correct the equipment position from the perspectives of mileage and coordinates, respectively. The mutual verification of the two types of correction results can filter out erroneous correction data caused by single identification anomalies, continuously suppressing position drift caused by long-distance travel, and steadily reducing the spatial distance between the equipment and the target defect.
[0019] 3. The railway defect location and automatic navigation method based on multimodal fusion positioning of the present invention calculates the cumulative mileage error after the equipment arrives at the landmark near the defect, thereby completing the overall compensation and calibration of the equipment's internal mileage coordinate system. After aligning the mileage benchmark with the actual track mileage, the sleeper identification weight is increased. The real-time sleeper number is obtained by superimposing the benchmark sleeper number with the on-track count. After the number matches the target parameters, the equipment is controlled to stop and lock the defect location. Mileage calibration eliminates the systematic mileage deviation accumulated by wheel wear, allowing subsequent sleeper number calculation to be carried out based on an accurate mileage benchmark. The sleeper physical number is used as the final positioning judgment basis, overcoming the accuracy shortcomings of simply relying on mileage to calculate the point. The counting compensation method in the turnout section can also be adapted to special sleeper layout conditions, steadily improving the positioning accuracy of the stop at the end of the defect. Attached Figure Description
[0020] Figure 1 This is a flowchart of a railway defect location and automatic navigation method based on multimodal fusion positioning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a navigation device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the system units according to an embodiment of the present invention; In all the accompanying drawings, the same reference numerals indicate the same technical features, specifically: 1-sleeper counting module, 2-control box, 3-display screen, 4-forward landscape vision module, 5-high-precision navigation module. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0022] Example 1 Please refer to Figure 1 This embodiment 1 provides a railway defect location and automatic navigation method based on multimodal fusion positioning, including: S1. Acquire defect data and salient feature data along the railway line, and construct a multimodal positioning and feature association database; the database includes defect feature vectors and salient feature feature vectors; both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values and sleeper physical numbers; the salient feature feature vectors also contain feature types and feature image data; S2. Control the navigation device to move towards the target defect location. During the movement, perform coordinated navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, collect real-time images of the landscape ahead and the sleepers below, identify the prominent features and sleepers currently being passed, and compare them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. S3. When the navigation device moves to a prominent feature adjacent to the target defect location, the current mileage output value of the navigation device is obtained; based on the current mileage output value and the standard operating mileage value of the prominent feature in the database, the mileage cumulative error is calculated; the mileage cumulative error is used to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. S4. After completing the internal mileage coordinate system compensation calibration, increase the weight of sleeper identification, and calculate the physical number of the sleeper currently being crossed based on the calibrated internal mileage coordinate system and the real-time identified sleepers; when the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, control the navigation device to stop, and complete the locking of the target defect location.
[0023] This embodiment 1 further elaborates on the above steps.
[0024] (1) Construction of related databases The core objective in constructing the multimodal positioning and feature association database is to systematically store railway line defect data and salient feature data, providing underlying data support for subsequent collaborative navigation and position correction. This database primarily contains two data structures: defect feature vectors and salient feature vectors. Both types of feature vectors include at least three basic positioning dimensions: BeiDou coordinates, operational mileage, and sleeper physical numbers. The salient feature vectors further include feature type and image data to facilitate subsequent visual matching and on-site verification.
[0025] The acquisition of underlying database data relies on a laser image detection system installed on the track inspection equipment. This system performs continuous automated inspection of the railway line, simultaneously acquiring 3D LiDAR data and high-definition image data. The 3D LiDAR is used to acquire high-precision 3D point cloud data of the railway line and track structure to reflect the spatial geometry and contour features of the equipment; the high-definition camera is used to acquire the texture and appearance details of the track surface and facilities along the line. Based on the acquired 3D LiDAR data and high-definition image data, defects are automatically identified and extracted. The location of defects is confirmed by analyzing the geometric deformation of the point cloud and the texture anomalies of the image, and further, the system acquires three types of positioning data corresponding to each defect: BeiDou location, sleeper number, and operating mileage. Simultaneously with defect data extraction, the system automatically extracts salient scene features data on the inspected line based on the same 3D LiDAR data and high-definition image data. Salient scene features refer to fixed reference objects along the railway line with obvious geometric or visual characteristics that are easily and stably identified by sensors, specifically including signal lights, catenary poles, bridge and tunnel locations, and curve features. For each type of prominent scene feature extracted, the system also obtains three types of positioning data: BeiDou location, sleeper number, and operating mileage.
[0026] Due to differences in sampling frequencies and data transmission delays among different sensors, a unified timestamp needs to be assigned to each defect data and each significant scene feature data. Based on this timestamp, combined with the real-time operating speed and pose information of the inspection vehicle, spatiotemporal alignment processing is performed on the 3D LiDAR data, high-definition image data, and various positioning data to eliminate time differences and spatial installation errors between sensors, ensuring that multi-source data from the same time and spatial location can be accurately mapped to a unified track coordinate system. After data alignment is completed, the system structurally encapsulates the aligned BeiDou location, sleeper number, and operating mileage of each defect to generate defect feature vectors, and encapsulates the aligned BeiDou location, sleeper number, operating mileage, feature type, and feature image data of each significant scene feature to generate significant feature vectors.
[0027] In the final stage of database construction, based on operating mileage and spatial distance, the feature vectors of defects are associated with the feature vectors of salient features in their vicinity. Through this encapsulation and association mechanism, it is ensured that each detected defect in the database corresponds to three types of positioning data: BeiDou location, sleeper number, and operating mileage, and is associated with salient feature data of salient features along the line, thus forming a complete multimodal defect-feature association topology.
[0028] (2) Cooperative navigation and correction During the process of controlling the navigation device to move towards the target defect location, the navigation device relies on its built-in hardware architecture to perform cooperative navigation and position correction tasks. Please refer to... Figure 2 The navigation device mainly comprises a control box, a forward landscape vision module, a sleeper counting module, and a high-precision navigation module. The forward landscape vision module is equipped with an area array camera, responsible for acquiring images of the forward landscape and identifying salient features. The sleeper counting module includes a bottom camera and an encoder, used to acquire images of the sleepers below and count them using encoder pulses. The control box integrates a synchronization module, an industrial computer, and an edge computing module. The synchronization module is responsible for triggering hardware-level time synchronization of the forward landscape vision module, sleeper counting module, and high-precision navigation module, ensuring strict alignment of multi-source data on a time reference. The edge computing module performs real-time inference and recognition on the acquired images and transmits the recognition results to the industrial computer for multimodal data fusion and navigation decision-making. During movement, collaborative navigation is performed based on BeiDou coordinates and operating mileage values to approach the target defect location. Simultaneously, real-time image acquisition identifies currently passed features and sleepers, comparing them with a multimodal positioning and feature association database to correct the real-time position.
[0029] The implementation of cooperative navigation relies on real-time evaluation of BeiDou signal quality indicators. When the quality indicators are higher than a preset threshold, it is determined that the current environment is open, and a fusion navigation mode that uses BeiDou positioning data as the primary source and odometer data as a secondary source is adopted to calculate the real-time position of the navigation equipment.
[0030] When the quality index is lower than or equal to a preset threshold, the BeiDou signal is deemed limited, such as in tunnels or bridge sections. In this case, the system automatically downgrades to a navigation mode based on odometer data. In downgraded mode, the odometer data is integrated and compensated using the heading angle and acceleration data output by the inertial measurement unit, thereby maintaining continuous and stable navigation output and avoiding position loss due to satellite signal obstruction.
[0031] Based on cooperative navigation, in this embodiment, the navigation device first executes a first-level position correction process based on salient features while traveling on the railway line. A front-end image acquisition device acquires real-time images of the foreground landscape, and performs feature extraction processing on these images to obtain the visual feature vectors of the current ground features. ,in A unique visual descriptor representing salient features within the current field of view. Simultaneously, the current estimated position is obtained from the joint calculation output of the navigation device's internal odometry and inertial measurement unit. ,in Includes the estimated mileage and estimated lateral offset at the current moment. Using the geometric center and a preset search radius To constrain the scope, a spatial search is performed in a multimodal localization and land cover association database to obtain a set of candidate salient land covers falling within the specified range. ,in Indicates the first One candidate significant feature, The index is a positive integer. Then, With sets The standard feature vector corresponding to each candidate land cover in the image Calculate the similarity score for each match by performing a similarity calculation. When a candidate feature Exceeding the preset matching threshold At that time, it is determined that the current salient feature has been successfully identified, and the standard operating mileage value of the candidate feature is stored in the database. As the true value, perform the first-level correction on the current estimated mileage of the navigation device, and output the mileage and position after the first-level correction. ,in This indicates the longitudinal mileage coordinates after anchoring to prominent features.
[0032] During the completion or parallel execution of the first-level correction, the second-level position correction process based on sleeper recognition is executed simultaneously. The acquisition of bottom sleeper images is not triggered at a fixed frequency, but rather based on the real-time speed of the navigation device. Adaptive adjustment of trigger frequency ,in and A positive correlation is observed to ensure that a spatially uniform sleeper image sequence can be acquired at different speeds. After each acquisition trigger, the pulse signal output from the wheel encoder is combined with... Perform timing verification, where The pulse count representing the number of times the wheel rotates within the acquisition interval between two adjacent image frames; if Beyond the present If a frame is found to have a corresponding reasonable pulse interval, it is determined that the frame is experiencing motion blur or spatial sampling distortion due to a sudden change in velocity, and is therefore discarded. Only valid sleeper images that pass the time-series verification are retained for subsequent processing. Feature extraction is performed on the valid sleeper images to obtain the current sleeper features. ,in This refers to the optical character recognition result of the physical number on the sleeper surface or a unique coded description of the sleeper texture. Subsequently, based on the database segment corresponding to the current estimated position of the navigation device, a continuous sequence of sleeper physical numbers within that segment is extracted. ,in Represents the first in the database Standard features of railway sleepers The index is a positive integer. A sliding window algorithm is used to... and Perform a bit-by-bit matching; when a match is successful, determine the standard spatial coordinates of the current sleeper in the database. ,in Includes the absolute three-dimensional geographic coordinates of the sleeper. Utilizing... Perform a second-level correction on the current original BeiDou coordinates of the navigation device, and output the BeiDou coordinates after the second-level correction. ,in This represents the global spatial coordinates after the sleepers are anchored.
[0033] In separate and After that, the consistency verification and fusion decision-making stage of the dual-source correction results is entered. First, the calculation is performed. The corresponding spatial location and The Euclidean distance between them is defined as the spatial deviation. ,in This characterizes the degree of inconsistency between the two-level independent correction results in a unified coordinate system. With preset safety threshold Comparison: When At that time, if the results of the two-level corrections are mutually corroborative and the data is reliable, a weighted fusion strategy is adopted to... and Perform optimal estimation fusion and output the updated real-time position of the navigation device. ,in As the initial state for the next cycle calculation; when If at least one level of correction result is found to have abnormal interference, the entire correction data is deemed unreliable, and the system automatically removes it. and No position update operation is performed. The calculated position before the correction is maintained as the navigation output until the next valid object triggers the first-level correction or the next valid sleeper triggers the second-level correction and passes the consistency check.
[0034] (3) Mileage compensation calibration When the navigation device moves to a prominent feature preceding the target defect location, it triggers a compensation calibration process for the internal mileage coordinate system to eliminate accumulated errors from long-distance travel. For example, when the navigation device travels on a railway line and passes prominent features sequentially, it first obtains the odometer interval measurement increment and the physical distance of the standard interval, and calculates the wheel diameter wear ratio coefficient. When the navigation device passes the preceding... When there is a prominent feature, record the cumulative mileage value output by the odometer at that time. ,in Indicates that the navigation device has reached the The internal reading of the odometer at each significant landmark; when the navigation device reaches the current 1st When there is a prominent feature, record the cumulative mileage value output by the odometer at that time. ,in Indicates that the navigation device has reached the The internal readings of the odometer at each salient feature are used. Based on these two readings, the odometer interval measurement increment between two adjacent salient features is calculated. ,in This characterizes the relative travel distance measured by the odometry between two adjacent salient features. Simultaneously, it retrieves the first [unit / item] from a multimodal localization and feature association database. Standard operating mileage of a prominent feature and the Standard operating mileage of a prominent feature And calculate the corresponding standard interval physical distance. ,in Characterizes the true, mapped, absolute physical length between two adjacent salient features. Subsequently, based on... and The ratio is used to calculate the real-time wheel diameter wear ratio coefficient of the odometer wheel. ,in A correction factor characterizing the current effective rolling circumference of the wheel relative to the calibrated circumference of the odometer, used to compensate for the change in distance per unit pulse caused by wheel wear.
[0035] While calculating the wheel diameter wear ratio coefficient, the system further calculates the absolute mileage drift error at the current node and performs zero-point compensation and underlying pulse equivalent update. Using the formula... Calculate the current number Absolute mileage drift error at prominent features ,in This represents the absolute deviation of the odometer's cumulative reading from the true standard mileage. The zero-point compensation is directly superimposed onto the current internal mileage coordinate system of the navigation device, making the calibrated internal mileage output value become... This eliminates all accumulated absolute deviations up to the current moment. Simultaneously, the calculated real-time wheel diameter wear ratio coefficient is used. The underlying pulse equivalent of the odometer is updated. The updated pulse equivalent is equal to the original calibrated pulse equivalent multiplied by [value missing]. This ensures that the displacement represented by each wheel encoder pulse during subsequent driving is converted according to the current actual wheel diameter, fundamentally correcting the mileage ratio calculation error caused by continuous wheel wear, and ensuring the accuracy of mileage increment measurement in the next interval.
[0036] After completing the geometric compensation of the internal mileage coordinate system, the system enters the dynamic adjustment stage of the multi-source fusion weights to adapt to the sensor reliability status after calibration. First, the first position covariance matrix output by the BeiDou positioning module before calibration is obtained. The second position covariance matrix calculated from the odometer output and the third location covariance matrix of the visual ground feature localization output ,in Characterizing the uncertainty distribution of BeiDou positioning results, Characterizing the uncertainty distribution of the odometer-estimated position, Characterizes the uncertainty distribution of visual feature matching and localization results.
[0037] When absolute mileage drift error The absolute value exceeds the preset error threshold. If the odometer was determined to be in a state of high error divergence before this calibration, and the reliability of its historical extrapolation data is low, then by increasing... The trace value is used to reduce the weight of odometry positioning in Kalman filter fusion, so that the fusion algorithm relies more on other high-precision observation sources in the current cycle.
[0038] Simultaneously, the sharpness index of the currently identified salient features is extracted. and matching confidence ,in A comprehensive evaluation value characterizing image focus quality and signal-to-noise ratio. Characterizes the reliability of the match between current ground features and database template features. Based on and Dynamic scaling The trace value, specifically, when and When both are at a high level, The trace value is compressed proportionally, thereby increasing the weight of visual ground feature localization in Kalman filter fusion. This allows the system to fully utilize high-quality visual observations to quickly and stably stabilize the fusion state during the transition window period when the odometer has just completed a large correction and its own confidence has not yet recovered, thus achieving smooth connection and accuracy maintenance of multimodal localization.
[0039] (4) Locating the location of the target disease In this embodiment, after completing the internal mileage coordinate system compensation calibration, the navigation device first executes a dynamic improvement process for sleeper identification weights to ensure that subsequent sleeper number estimation has a high-precision fusion positioning basis. The residual error estimate of the internal mileage coordinate system after compensation calibration is then obtained. ,in The remaining uncorrected deviation between the odometer output and the actual position after the calibration operation is completed is characterized; simultaneously, the real-time acquired bottom sleeper image is processed to obtain the recognition confidence level of the current frame sleeper image. ,in This characterizes the reliability of the sleeper feature extraction and database matching results. Based on Constructing the odometer observation noise covariance matrix ,in The diagonal element values and The square of the equation is positively correlated and used to quantify the uncertainty of current odometer observations; based on Constructing the noise covariance matrix of sleeper visual observation ,in The diagonal element values and They are negatively correlated and used to quantify the uncertainty of current visual observations of sleepers. and As measurement update parameters, they are input to the extended Kalman filter. When Less than the preset convergence threshold and Greater than the preset confidence threshold If it is determined that the current period is within the high-precision window of the odometer and the sleeper identification result is reliable, then the range is narrowed. The diagonal element values automatically increase the weight coefficients corresponding to the sleeper observation components in the extended Kalman filter gain matrix, causing the fusion state estimation results to converge towards the sleeper identification results, thereby dynamically improving the sleeper identification weights.
[0040] After improving the sleeper identification weights and obtaining a stable fusion positioning output, the calculation process for the current sleeper physical numbers is further executed. This involves calculating the sleeper physical numbers corresponding to nearby salient features that have undergone compensation calibration. As the benchmark starting point, where This represents the absolute sleeper number that has been confirmed and uniquely identified in the database at the location of the prominent feature. Starting from the location of this prominent feature, the number of valid sleepers identified in real-time by the navigation equipment as it travels to the current location is counted. ,in This represents the cumulative increase in the number of sleepers detected from the baseline to the current time. (Using the formula...) Calculate the current physical number of the sleeper ,in This indicates the absolute physical number in the database of the sleeper that the navigation device is currently crossing or has just crossed. When the navigation device is detected to be in a turnout section, a sleeper spacing correction factor for the turnout section is introduced. right Compensation will be provided, including This characterizes the proportion of the number of sleepers per unit mileage relative to the standard section, caused by changes in sleeper type and uneven spacing within the turnout section; the compensated increase in the number of sleepers is... and will This serves as a formula for calculating the physical numbering of sleepers within turnout sections, aiming to eliminate errors in numbering calculations caused by uneven sleeper distribution at turnout locations.
[0041] Continuously calculate and output the current sleeper physical number. During the process, the location of the target defect is simultaneously determined. The physical number of the target sleeper corresponding to the location of the defect to be detected is retrieved in advance from the defect management database. ,in This represents the unique sleeper anchor point number bound to the database for this defect. The data will be calculated in real time. and Perform frame-by-frame comparison: when and When the target defect location is reached, the navigation device is determined to have arrived at the location of the target defect. A stop control command is immediately generated and sent to the motion control unit of the navigation device, which controls the navigation device to stop smoothly at the current sleeper position, thus completing the locking of the target defect location.
[0042] Example 2 Please refer to Figure 3 This embodiment 2 provides a railway defect location and automatic navigation system based on multimodal fusion positioning, including: A relational database construction unit is used to acquire railway defect data and salient feature data to construct a multimodal positioning and feature association database. The database includes defect feature vectors and salient feature feature vectors. Both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values, and sleeper physical numbers. The salient feature feature vectors also contain feature types and feature image data. The collaborative navigation and correction unit is used to control the navigation device to move towards the target defect location. During the movement, it performs collaborative navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, it collects images of the landscape ahead and the sleepers below in real time, identifies the salient features and sleepers that are currently being passed, and compares them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. The mileage compensation calibration unit is used to obtain the current mileage output value of the navigation device when the navigation device moves to a nearby prominent feature preceding the target defect location; calculate the mileage accumulation error based on the current mileage output value and the standard operating mileage value of the nearby prominent feature in the database; and use the mileage accumulation error to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. The target defect location locking unit is used to increase the weight of sleeper identification after the internal mileage coordinate system compensation calibration is completed. Based on the calibrated internal mileage coordinate system and the real-time identified sleepers, it calculates the physical number of the sleeper currently being crossed. When the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, it controls the navigation device to stop and completes the locking of the target defect location.
[0043] Example 3 This embodiment 3 also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement any step of a railway defect location and automatic navigation method based on multimodal fusion positioning.
[0044] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0045] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.
[0046] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for railway defect localization and automatic navigation based on multimodal fusion positioning, characterized in that, include: S1. Acquire defect data and salient feature data along the railway line, and construct a multimodal positioning and feature association database; the database includes defect feature vectors and salient feature feature vectors; both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values and sleeper physical numbers; the salient feature feature vectors also contain feature types and feature image data; S2. Control the navigation device to move towards the target defect location. During the movement, perform coordinated navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, collect real-time images of the landscape ahead and the sleepers below, identify the prominent features and sleepers currently being passed, and compare them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. S3. When the navigation device moves to a prominent feature adjacent to the target disease location, the current mileage output value of the navigation device is obtained; Based on the current mileage output value and the standard operating mileage value of the nearby significant features in the database, the mileage cumulative error is calculated; the mileage cumulative error is used to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. S4. After completing the internal mileage coordinate system compensation calibration, increase the weight of sleeper identification, and calculate the physical number of the sleeper currently being crossed based on the calibrated internal mileage coordinate system and the real-time identified sleepers; when the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, control the navigation device to stop, and complete the locking of the target defect location.
2. The railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S1, the specific process of constructing the multimodal positioning and ground feature association database includes: The laser image detection system is used to automatically detect railway lines on-site, and to collect three-dimensional lidar data and high-definition image data of the railway lines. Based on the three-dimensional lidar data and the high-definition image data, the defects are identified and extracted, and three types of positioning data are obtained for each defect: Beidou location, sleeper number, and operating mileage. Based on the three-dimensional lidar data and the high-definition image data, the system automatically extracts data on significant scene features on the detection line. The data on significant scene features includes signal lights, catenary poles, bridge and tunnel locations, and curve features. The system also obtains three types of positioning data for each significant scene feature: BeiDou location, sleeper number, and operating mileage. A unified timestamp is assigned to each disease data and each significant scene feature data, and the three-dimensional lidar data, high-definition image data and positioning data are spatiotemporally aligned based on the timestamp; The BeiDou location, sleeper number, and operating mileage of each aligned defect are encapsulated to generate the defect feature vector. The BeiDou location, sleeper number, operating mileage, feature type, and feature image data of each aligned salient scene feature are encapsulated to generate the salient feature feature vector. This ensures that each detected defect corresponds to three types of positioning data: BeiDou location, sleeper number, and operating mileage, as well as associated salient scene feature data of the line.
3. The railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S2, collaborative navigation based on the BeiDou coordinates and the operational mileage value is performed to approximate the target defect location, specifically including: Real-time assessment of BeiDou signal quality indicators; When the quality index is higher than the preset threshold, the real-time position of the navigation device is calculated based on BeiDou positioning data as the primary source and odometer data as the secondary source. When the quality index is lower than or equal to the preset threshold, the BeiDou signal is determined to be restricted, and the system automatically downgrades to a navigation mode based on odometer data. The system then uses the heading angle and acceleration data output by the inertial measurement unit to perform integral compensation on the odometer data to maintain continuous navigation in tunnel or bridge sections.
4. The railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S2, real-time acquisition of images of the foreground landscape and the sleepers below is performed to identify prominent features and sleepers currently being passed, specifically including: The system acquires real-time images of the foreground landscape and extracts the visual features of current ground features; obtains the current estimated position of the navigation device, and retrieves candidate salient features from the multimodal positioning and ground feature association database with the estimated position as the center and a preset search radius; performs similarity matching between the visual features of the current ground features and the ground feature image data of the candidate salient features; when the matching similarity exceeds a preset threshold, the identification is deemed successful, and the current estimated mileage of the navigation device is corrected to the first level using the standard operating mileage value of the identified salient features in the database; Bottom sleeper images are acquired at a trigger frequency that adapts to the speed of the navigation device, and time-series verification is performed in conjunction with the pulse signals output by the wheel encoders to eliminate redundant or invalid images caused by sudden speed changes. Feature extraction is performed on the verified sleeper images to obtain the surface physical number or texture features of the current sleeper. The obtained features are matched with the sleeper physical number sequence of the corresponding section in the database using a sliding window. When the match is successful, the current BeiDou coordinates of the navigation device are corrected using the standard spatial coordinates of the sleeper in the database. Calculate the spatial deviation between the first-level corrected mileage position and the second-level corrected BeiDou coordinates; when the spatial deviation is less than a preset safety threshold, merge the correction results of the two to update the real-time position of the navigation device; when the spatial deviation is greater than or equal to the safety threshold, determine that the current identification result is abnormal, remove the correction data, and maintain the estimated position of the navigation device before correction to continue driving until the next valid object or sleeper triggers the correction.
5. The railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S3, the internal mileage coordinate system of the navigation device is compensated and calibrated using the accumulated mileage error, specifically including: Obtaining navigation device information after the preceding steps Odometry output value at a prominent feature And the current number Odometry output value at a prominent feature ; Calculate the odometry interval measurement increment between two adjacent salient features. And obtain the corresponding standard interval physical distance in the multimodal positioning and ground feature association database. ; Based on the above and The ratio of the two values is used to calculate the real-time wheel diameter wear ratio coefficient of the odometer wheel; simultaneously, the formula is used... Calculate the absolute mileage drift error of the current node. ; The absolute mileage drift error As a zero-point compensation, it is directly superimposed on the current internal mileage coordinate system of the navigation device to eliminate the cumulative absolute deviation; The real-time wheel diameter wear ratio coefficient is used to update the underlying pulse equivalent of the odometer in order to correct the mileage ratio calculation error caused by wheel wear during subsequent driving. Obtain the first position covariance matrix output by the BeiDou positioning module before calibration, the second position covariance matrix calculated by the odometer, and the third position covariance matrix output by the visual ground feature positioning. When the absolute mileage drift error When the preset error threshold is exceeded, the odometer is determined to be in a high error divergence state. The weight of the odometer positioning in the Kalman filter fusion is reduced by increasing the trace value of the second position covariance matrix. Simultaneously, the sharpness index and matching confidence of the currently identified salient ground features are extracted, and the trace value of the third location covariance matrix is dynamically reduced based on the sharpness index and matching confidence to improve the fusion weight of visual ground feature localization.
6. The railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S4, after completing the internal mileage coordinate system compensation calibration, the weight of sleeper identification is increased, specifically including: Obtain the residual error estimate of the internal mileage coordinate system after compensation and calibration, as well as the recognition confidence of the real-time acquired bottom sleeper images; Based on the residual error estimate, an odometer observation noise covariance matrix is constructed, and based on the identification confidence, a sleeper visual observation noise covariance matrix is constructed. The odometer observation noise covariance matrix and the sleeper visual observation noise covariance matrix are input into the extended Kalman filter as measurement update parameters; When the residual error estimate is less than the preset convergence threshold and the recognition confidence is greater than the preset confidence threshold, the diagonal element values of the sleeper visual observation noise covariance matrix are reduced, and the weight coefficients corresponding to the sleeper observation components in the extended Kalman filter gain matrix are increased, so that the fusion state estimation result converges to the sleeper recognition result, thereby completing the dynamic improvement of the sleeper recognition weight.
7. A railway defect location and automatic navigation method based on multimodal fusion positioning according to claim 1, characterized in that, In step S4, calculating the physical number of the sleeper currently being crossed specifically includes: The physical sleeper numbers corresponding to the nearby prominent features for which compensation calibration has been completed. As the benchmark starting point; Count the number of sleepers identified in real time from the nearby prominent features to the current location. ; Through formula Calculate the current physical number of the sleeper ; When the navigation device is detected to be in a turnout section, a turnout section sleeper spacing correction coefficient is introduced for the number of sleepers. Compensation will be provided to eliminate errors in numbering calculations caused by variations in sleeper type and uneven spacing at turnouts.
8. A railway defect location and automatic navigation method based on multimodal fusion positioning according to any one of claims 1-7, characterized in that, The navigation device includes a control box, a forward landscape vision module, a sleeper counting module, and a high-precision navigation module; The foreground landscape vision module includes an area array camera, used to acquire foreground landscape images and identify prominent scene features; The sleeper counting module includes a bottom camera and an encoder, which are used to acquire images of the bottom sleepers and count the sleepers by combining encoder pulses. The control box includes a synchronization module, an industrial computer, and an edge computing module; the synchronization module is used to trigger hardware-level time synchronization of the forward landscape vision module, the sleeper counting module, and the high-precision navigation module. The edge computing module is used to perform real-time reasoning and recognition on the acquired images, and transmit the recognition results to the industrial control computer for multimodal data fusion and navigation decision-making.
9. A railway defect location and automatic navigation system based on multimodal fusion positioning, characterized in that, include: A relational database construction unit is used to acquire railway defect data and salient feature data to construct a multimodal positioning and feature association database. The database includes defect feature vectors and salient feature feature vectors. Both defect feature vectors and salient feature feature vectors contain at least BeiDou coordinates, operating mileage values, and sleeper physical numbers. The salient feature feature vectors also contain feature types and feature image data. The collaborative navigation and correction unit is used to control the navigation device to move towards the target defect location. During the movement, it performs collaborative navigation based on the Beidou coordinates and the operating mileage value to approach the target defect location. At the same time, it collects images of the landscape ahead and the sleepers below in real time, identifies the salient features and sleepers that are currently being passed, and compares them with the multimodal positioning and feature association database to correct the real-time position of the navigation device. The mileage compensation calibration unit is used to obtain the current mileage output value of the navigation device when the navigation device moves to a prominent feature adjacent to the target disease location. Based on the current mileage output value and the standard operating mileage value of the nearby significant features in the database, the mileage cumulative error is calculated; the mileage cumulative error is used to compensate and calibrate the internal mileage coordinate system of the navigation device so that the internal mileage coordinate system is aligned with the actual physical mileage of the track. The target defect location locking unit is used to increase the weight of sleeper identification after the internal mileage coordinate system compensation calibration is completed. Based on the calibrated internal mileage coordinate system and the real-time identified sleepers, it calculates the physical number of the sleeper currently being crossed. When the calculated physical number of the sleeper is consistent with the physical number of the target sleeper corresponding to the target defect location, it controls the navigation device to stop and completes the locking of the target defect location.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8: a method for locating railway defects and providing automatic navigation based on multimodal fusion positioning.
