Multi-modal fusion dynamic snakelike convolution rail fat edge detection method and multi-modal fusion dynamic snakelike convolution rail fat edge detection system

Abstract

The invention belongs to the field of railway track detection, and discloses a multi-modal fusion dynamic snakelike convolution steel rail fat edge detection method and system, and the method comprises the steps: obtaining multi-source data, including steel rail three-dimensional data, satellite positioning data and inspection vehicle attitude information; processing the three-dimensional data of the steel rail, extracting contour features of a fat edge of the steel rail, and outputting a mask and three-dimensional size estimation of a fat edge region; performing time sequence modeling on the satellite positioning data to obtain spatial position information of the inspection vehicle; based on the attitude information of the inspection vehicle, fusing the mask of the fat edge region and the three-dimensional size estimation with the spatial position information of the inspection vehicle, and calculating a space-time consistency index; and carrying out global optimization on the fused fat edge three-dimensional size and spatial position by adopting a self-adaptive weighted optimization algorithm and taking a space-time consistency index as a weight basis, and outputting a steel rail fat edge detection and positioning result. The invention provides a steel rail fat edge detection scheme which is high in detection precision, accurate in spatial positioning and high in robustness.

Description

Technical Field

[0001] This invention relates to the field of railway track inspection technology, and in particular to a multimodal fusion dynamic serpentine convolution rail fat edge detection method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Rail edge thickening is a common surface geometric defect in railway tracks. It mainly manifests as plastic flow of the rail gauge angle and side metal material under repeated wheel-rail loads, forming raised edges distributed along the track direction. This defect worsens wheel-rail contact, increases impact vibration, and can even lead to safety problems. Therefore, efficient and accurate detection and location of rail edge thickening are crucial.

[0004] Existing inspection methods mainly rely on manual inspection or automated inspection based on a single sensor. Manual inspection is inefficient and highly subjective. Automated inspection methods based on lasers, profilometers, or ultrasound are mostly for localized defects and are difficult to accurately characterize geometric defects such as thick edges that are distributed along the track direction and have slender and complex shapes, which can easily lead to missed detections, false detections, or inaccurate dimensional assessments.

[0005] In recent years, machine vision and 3D reconstruction technologies have made some progress, but there are still shortcomings: First, traditional convolution operators are difficult to adaptively fit the contours of thick, thin, and curved lines; second, the detection results usually lack a stable mapping with the mileage and geographical coordinates, making it difficult to use them directly for maintenance work; finally, the degree of multi-sensor information fusion is low, there is a lack of a unified quality control mechanism, and the robustness is poor under complex working conditions. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a multimodal fusion dynamic serpentine convolution method and system for detecting rail ballast, providing a rail ballast detection scheme with high detection accuracy, precise spatial positioning, and strong robustness.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multimodal fusion dynamic serpentine convolution method for detecting rail thick edges, comprising the following steps: To obtain multi-source data required for rail edge detection, the multi-source data includes three-dimensional rail data, satellite positioning data, and inspection vehicle attitude information; A dynamic serpentine convolutional network is used to process the 3D data of the rail, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region. Temporal convolutional networks are used to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; Based on the inspection vehicle's attitude information, the mask and 3D size estimation of the fat edge region are fused with the inspection vehicle's spatial position information, and the spatiotemporal consistency index is calculated. An adaptive weighted optimization algorithm is adopted, using the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail fat edge, and output the rail fat edge detection and positioning results.

[0008] As an alternative implementation, the dynamic serpentine convolutional network is a DS-ConvLSTM network, whose convolutional kernel sampling points can be adaptively distributed in a serpentine pattern according to the slender curved profile of the rail fat side, and optimized through learnable offsets and geometric energy constraints to make the convolutional kernel closely follow the boundary of the fat side.

[0009] As an alternative implementation, the temporal convolutional network is a TCN-LSTM network, which extracts local and high-frequency features of satellite positioning data through the TCN network and models the long-term temporal dependence of rail position through the LSTM network. The loss function used during network training includes both position error and clock / phase difference constraints.

[0010] As an alternative implementation method, multi-source data is synchronized through hardware triggering and unified timestamps. Specifically, the PPS pulse signal output by the satellite positioning receiver is used to trigger the exposure of the structured light camera and the sampling of the IMU, and the time synchronization algorithm is used to achieve microsecond-level time alignment.

[0011] As an alternative implementation method, the spatiotemporal consistency index is calculated using the following formula: ; in, For the first The intersection-union ratio of the frame fat edge mask and the annotation mask. This represents the geographic coordinate error of the center of the fat edge in this frame. This represents the displacement residual along the track mileage direction. , The normalization coefficient is... λ 1, λ 2, λ 3 represents the weighting coefficient.

[0012] As an alternative implementation method, the adaptive weighted optimization algorithm is the adaptive weighted least squares method, and its objective function is: ; in, For the first The truth vector of a frame. For system output, For the parameters to be estimated, the weights are... Determined by joint adaptive analysis of spatiotemporal consistency index and residuals: ; in, An index to enhance the influence of highly consistent samples, To prevent small constants with a denominator of zero.

[0013] Secondly, the present invention provides a multimodal fusion dynamic serpentine convolution rail thick edge detection system, comprising: The data acquisition module is configured to acquire multi-source data required for rail edge detection, including rail 3D data, satellite positioning data, and inspection vehicle attitude information. The preliminary detection module is configured to: process the 3D data of the rail using a dynamic serpentine convolutional network, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region; The spatial positioning module is configured to: use a temporal convolutional network to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; The information fusion module is configured to: based on the inspection vehicle's attitude information, fuse the mask and 3D size estimation of the fat edge region with the spatial position information of the inspection vehicle, and calculate the spatiotemporal consistency index; The rail edge detection module is configured to use an adaptive weighted optimization algorithm, with the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail edge, and output the rail edge detection and positioning results.

[0014] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0016] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a multimodal fusion dynamic serpentine convolutional rail thick edge detection method and system. Employing a dynamic serpentine convolutional network, it adaptively fits the complex contours of thick rail edges, which are long, curved, and burr-laden, resulting in accurate feature extraction and significantly improved accuracy in contour extraction and size estimation. By using a TCN-LSTM network to perform temporal modeling of BeiDou carrier phase and combining it with double-difference processing, the positioning accuracy is improved to the centimeter level, achieving precise spatial positioning of thick edge defects. A spatiotemporal consistency index (STCI) is proposed to comprehensively evaluate the quality of detection results. Based on this, an adaptive weighted least squares (AWLS) method is used for global optimization, effectively suppressing abnormal sample interference and improving the system's robustness and overall estimation accuracy under complex working conditions. A point cloud visualization system is constructed to integrate and display the detection results with positioning information and confidence levels (mapped via STCI), intuitively presenting the location, size, and confidence level of defects.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is an architecture diagram of the multimodal fusion dynamic serpentine convolution rail thick edge detection system of the present invention; Figure 2 This is a data processing flowchart of the multimodal fusion dynamic serpentine convolution rail thick edge detection method of the present invention; Figure 3 The following are simulation results of the fat edge detection of the present invention from different perspectives: (a) is the original point cloud side view of the rail and fat edge obtained by structured light reconstruction; (b) is the detection result of the fat edge mask superimposed on the point cloud after processing by the DS-ConvLSTM network; and (c) is the visualization result of the three-dimensional size and spatial position of the fat edge after fusing positioning information and performing AWLS global optimization. Figure 4 This is a point cloud display diagram for the fat edge detection of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed description is exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Terminology Explanation: 1. Rail Edge Thickening: Rail edge thickening refers to a type of surface geometric defect caused by the plastic flow of metal on the rail gauge angle and rail head side under long-term wheel-rail interaction. This results in protrusions, curling, or thickened edges distributed along the track direction on the rail head side, causing the rail cross-sectional geometry to deviate from the design profile. This invention mainly targets the automatic detection, three-dimensional dimensional assessment, and spatial positioning of the aforementioned edge thickening defect.

[0026] 2. BeiDou High-Precision Positioning System (BDS): The BeiDou High-Precision Positioning System is a global satellite navigation system independently constructed and operated by my country, providing services such as position, velocity, and time. This invention utilizes a multi-frequency BeiDou receiver supporting carrier phase differential to acquire the geodetic coordinates and time information of the track inspection vehicle, and combines differential processing of the reference station with carrier phase dual-difference processing to achieve centimeter-level track positioning accuracy.

[0027] 3. Structured Light Camera: A structured light camera is an imaging device that reconstructs the three-dimensional shape of an object by projecting a specific structured light pattern and capturing its deformation on the surface of the object being measured. This invention uses line structured light or stripe structured light to perform non-contact scanning of the rail surface, acquiring the rail cross-sectional profile and three-dimensional point cloud of the rail head side, providing high-resolution data for extracting the geometric features of the rail edge.

[0028] 4. Inertial Measurement Unit (IMU): An inertial measurement unit typically consists of an accelerometer and a gyroscope, used to measure the angular velocity and linear acceleration of the vehicle. This invention utilizes the IMU to acquire the attitude change information of the track inspection vehicle, jointly calculates the vehicle's trajectory with the BeiDou positioning results, and achieves accurate conversion between the vehicle coordinate system, the track coordinate system, and the geographic coordinate system through external parameter calibration and rigid body transformation.

[0029] 5. Dynamic Snake Convolution and DS-ConvLSTM Network: Dynamic snake convolution is a deformable convolution operator designed for slender, curved boundary structures. It controls the sampling position of the convolution kernel by parameterizing the snake path and learningable offsets, causing it to be distributed in a snake-like pattern along the target edge. Geometric smoothing and edge gradient constraints are introduced to enhance sensitivity to irregular boundaries. This invention uses dynamic snake convolution as the core operator of the encoder in the DS-ConvLSTM network, combined with LSTM temporal units to model the thick edge deformation between consecutive scanning frames, achieving joint prediction of the thick edge mask and 3D dimensions of the rail.

[0030] 6. TCN-LSTM Network: The TCN-LSTM network consists of a Temporal Convolutional Network (TCN) and a Long Short-Term Memory (LSTM) network connected in series. The TCN extracts local and high-frequency features of the BeiDou carrier phase sequence through dilated convolution, while the LSTM is used to model the long-term temporal dependence of the orbital position. This invention utilizes the TCN-LSTM network to jointly model BeiDou and differential observations, and introduces position error and clock / phase difference constraints into the loss function to improve orbital positioning accuracy and robustness.

[0031] 7. Spatio-Temporal Consistency Index (STCI): The Spatio-Temporal Consistency Index (STCI) is a comprehensive index proposed in this invention to measure the quality of fat edge detection results. It quantifies the consistency and reliability of single-frame results in three dimensions: "morphology, spatial location, and time series" by weighting and combining factors such as the cross-union ratio between the fat edge mask and the manually labeled mask, the geographical coordinate error of the detection point, and the mileage continuity along the route. This provides a unified evaluation basis for subsequent AWLS optimization and visualization confidence expression.

[0032] 8. Adaptive Weighted Least Squares (AWLS): Adaptive Weighted Least Squares (AWLS) is a parameter estimation method that introduces adaptively varying weights based on sample quality within the traditional least squares framework. In this invention, AWLS uses the STCI index and residual size to jointly determine the weight of each frame's samples, assigning greater weights to samples with high consistency and small residuals, and reducing the weights of anomalous or noisy samples, thereby achieving robust optimization of the estimation results for the 3D size and spatial location of fat edges.

[0033] Example 1 like Figures 1 to 4As shown, this embodiment provides a multimodal fusion-based dynamic serpentine convolution method for detecting rail thick edges, including the following steps: To obtain multi-source data required for rail edge detection, the multi-source data includes three-dimensional rail data, satellite positioning data, and inspection vehicle attitude information; A dynamic serpentine convolutional network is used to process the 3D data of the rail, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region. Temporal convolutional networks are used to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; Based on the inspection vehicle's attitude information, the mask and 3D size estimation of the fat edge region are fused with the inspection vehicle's spatial position information, and the spatiotemporal consistency index is calculated. An adaptive weighted optimization algorithm is adopted, using the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail fat edge, and output the rail fat edge detection and positioning results.

[0034] The specific solution of the present invention is as follows: Step 1: Data Acquisition Module Construction: In implementing this invention, a high-performance data acquisition module needs to be constructed first, including high-precision locators and sensors. First, a positioning receiver and various sensors are installed on the track inspection vehicle. The receiver acquires BeiDou positioning information, and the track inspection vehicle detects rail information, recording its movement trajectory in real time and uploading it to the terminal. Simultaneously, various sensors, such as a structured light camera, collect the geometric and positional parameters of the rails in real time. The data acquisition module is responsible for transmitting this data to the subsequent processing module.

[0035] Step 2: Implementation of the testing module: Using real equipment and materials, simulate rail edge thickening is performed to test the effect of simulated rail edge thickening from different perspectives. This simulated testing can predict and evaluate the performance and effect of rail edge thickening without actual testing; and the parameters can be flexibly adjusted to quickly verify multiple schemes and accelerate the actual rail edge thickening testing process.

[0036] Step 3: Information Processing Module Construction: The information processing module employs a neural network system, including the conversion and processing of rail bulge information from BeiDou positioning, and the detection of geometric parameters of the rail. It records the horizontal position scanned by the equipment and analyzes the length of the bulge. After obtaining the processing results, the manually processed data is compared with the actual rail measurement data to analyze the error and verify the accuracy.

[0037] Step 4: Image Display Module Construction: Construct an integrated system for visualizing and positioning rail point clouds. Display key information such as the current position and error ellipse in real time within the point cloud scene. Highlight and render detected thick-edge areas, associating them with their geographic coordinates.

[0038] Preferably, the details of the construction method for the data acquisition module in step 1 are as follows: S1-1: Sensor selection.

[0039] A high-precision BeiDou receiver supporting multi-frequency carrier phase observation and PPS pulse output was selected to ensure centimeter-level positioning capability even in the track environment. A high-resolution structured light camera with adjustable line / striped structured light modes and high-speed acquisition capability was selected to capture the subtle geometric features of the rail head and sides. A six-axis IMU (including a three-axis accelerometer and a three-axis gyroscope) was selected to measure the attitude changes and linear acceleration of the track inspection vehicle, providing an inertial information basis for multi-sensor fusion.

[0040] S1-2: Installation and camera calibration.

[0041] The BeiDou receiver is fixedly mounted on a rigid structure of the inspection vehicle or mobile platform and connected to the onboard industrial control computer via a data cable. Communication parameters and data output formats are configured. The structured light camera is installed in a field of view that completely covers the rail head and rail edge area, ensuring a stable viewing angle during train operation / inspection. The internal and external parameters of the structured light camera are calibrated using a calibration board; its imaging model is a pinhole model. ; in, Let be the homogeneous coordinates of a point on the rail surface in the world coordinate system. These are pixel coordinates; For the camera intrinsic parameter matrix, This is the camera extrinsic parameter matrix.

[0042] The rotation matrix in the external parameters is dynamically corrected using the angular velocity measured by the IMU. Let the first... The angular velocity at each sampling time is Sampling interval is Then we have: ; in, This is the rotation matrix of the sensor carrier coordinate system (IMU coordinate system) relative to a reference coordinate system (such as the vehicle coordinate system). An operator is used to convert a three-dimensional vector into a skew-symmetric matrix, which is used to implement the exponential mapping on the SO(3) Lie group, thereby compensating for the influence of the vehicle's attitude change on the structured light measurement results.

[0043] S1-3: Sensor periodic calibration and health monitoring.

[0044] Establish a periodic calibration system for BeiDou receivers, structured light cameras, and IMUs, and verify the measurement deviations of each sensor through static calibration and short-range dynamic calibration; set up equipment health check procedures to alarm and reject situations such as abnormal signal-to-noise ratio, data loss, and abnormal timestamps, so as to ensure the reliability of the collected data.

[0045] S1-4: Time synchronization and hardware triggering.

[0046] High-precision timestamps and PPS pulse signals output from a BeiDou receiver are used to synchronize the time of the structured light camera and the IMU. Let the structured light camera timestamp be... Beidou timestamp is The system time deviation is The synchronization time is defined as follows: ; in The edge differences of multiple PPS sets were obtained by fitting the least squares method: ; The Beidou receiver outputs PPS pulses to simultaneously trigger the structured light camera's exposure and the IMU's sampling, achieving hardware-level synchronization.

[0047] S1-5: Acquisition end buffering and preprocessing.

[0048] The data acquisition module has built-in cache and preprocessing units to perform preliminary denoising, cropping and format conversion on the original structured light images and point clouds, and to sort and interpolate the BeiDou and IMU data by time to generate data frames with unified timestamps, providing an interface for subsequent batch processing and real-time processing.

[0049] S1-6: High-speed communication link.

[0050] The acquisition end is connected to the information processing unit using Gigabit Ethernet or high-speed serial bus, and the data integrity and real-time performance are ensured by using data queues and retransmission mechanisms while the inspection vehicle is in motion.

[0051] Preferably, the implementation details of step 2 for the detection test module are as follows: S2-1: Disease simulation.

[0052] Based on the geometric parameters of the thickened rail edge obtained from on-site statistics, multiple models of thickened rail edge defects were designed, including models with different heights, widths, lengths, and local burr morphologies. Rail specimens can be fabricated using a combination of 3D printing and metal processing to make their reflective properties and geometry as close as possible to those of real thickened rail edges.

[0053] S2-2: Environmental disturbance simulation.

[0054] Various lighting conditions (direct light, diffused light, low illumination, etc.) were set up in the experimental environment, and dust, oil, and rust were applied to the surface of the specimen to simulate the pollution state and surface roughness changes of the rails in the field.

[0055] S2-3: Multi-condition data acquisition.

[0056] Under the aforementioned combined conditions of defects and environmental factors, the data acquisition module is operated to collect 3D point cloud data of the rails, BeiDou positioning data, and IMU attitude information. By controlling the inspection vehicle's operating speed and track orientation, a dataset covering multiple perspectives, speeds, and operating conditions is generated to ensure the diversity of training samples.

[0057] S2-4: Structured light 3D reconstruction and projection model.

[0058] Three-dimensional reconstruction is performed based on the fringe images acquired by a structured light camera. Let the light intensity distribution of the fringes along the local x-axis on the rail surface be... ; in Background light intensity, In order to adjust the system, As the initial phase, the fringe frequency is: ; in The width of the camera's field of view. The desired number of stripes is preferred. For features with thick, thin, and convex edges, a larger number is preferred. This makes the phase distortion in the fat edge region obvious, which is convenient for dynamic serpentine convolution to extract local geometric deformation.

[0059] S2-5: Calculation of performance evaluation indicators.

[0060] Based on the reconstruction results obtained from S2-4 and the calibrated "true value" data, the accuracy of disease type identification, the accuracy of disease location, and the size error are calculated.

[0061] Define the weighted F1 index (WF1) to evaluate the performance in identifying multiple disease categories: ; in This represents the total number of disease categories. , , Categories The number of true positive, false positive, and false negative samples. As a category weight, a larger weight can be set for diseases caused by fertile edges to highlight the detection performance of fertile edges.

[0062] The positioning error in the image coordinate system is defined as: ; in The coordinates of the disease center pixel predicted by the network. These are the actual center pixel coordinates, manually labeled.

[0063] Positioning errors in geographic coordinate systems are calculated by projecting the predicted and actual points onto a geographic coordinate system (such as WGS-84 or a projected coordinate system) to determine the spherical / planar distance.

[0064] Dimensional errors are expressed as mean absolute percentage error (MAPE) on a three-dimensional scale: ; in , , For the first The actual length, width, and height of each disease. , , This is an estimated value for the present invention.

[0065] S2-6: Parameter optimization and cross-validation.

[0066] Based on indicators such as WF1, positioning error, and size error, the scanning frequency, resolution, exposure time, BeiDou sampling frequency, and filtering parameters of the structured light camera are jointly optimized. The optimal parameter combination and network structure are selected using cross-validation to improve the generalization ability of the model.

[0067] S2-7: Verification through multiple rounds of testing.

[0068] After parameter optimization, disease simulation and environmental interference simulation tests were conducted again to verify the stability and robustness of the system under different operating conditions, ensuring that it can maintain high detection performance in actual line applications.

[0069] Preferably, the details of the construction method for the information processing module in step 3 are as follows: S3-1: Coordinate transformation and spatial alignment.

[0070] Based on the BeiDou geodetic coordinates obtained from S1-5, they are converted to a geocentric-fixed coordinate system (ECEF) or a local projected coordinate system. Taking WGS-84 as an example, the latitude is... Longitude is The earth is high The semi-major axis of the ellipsoid is The first eccentricity is radius of curvature ,but: ; ; ; Using the rigid body transformation matrix obtained from calibration: ; in The coordinates of the structured light point cloud are... For rotation matrix, As a translation vector, the structured light point cloud coordinates are transformed to a global coordinate system consistent with the BeiDou coordinate system, achieving spatial alignment between the two. The structured light point cloud coordinates are the point cloud coordinates in the structured light camera coordinate system. They are calculated by calibrating the intrinsic and extrinsic parameters of the structured light camera and performing phase decoding and triangulation on the acquired structured light fringe images, resulting in the three-dimensional point coordinates of the rail surface.

[0071] S3-2: Point cloud preprocessing.

[0072] The 3D point cloud acquired by the structured light camera is subjected to denoising and filtering. Statistical filtering, radius filtering, and median filtering algorithms are used to remove isolated points and significant noise. Non-rail surface areas such as sleepers and fasteners are segmented, while retaining the point cloud related to the rail and its edge. A local coordinate system is established based on the rail direction and normal direction, and the point cloud is normalized.

[0073] S3-3: Dynamic snake convolution and DS-ConvLSTM fat edge feature extraction.

[0074] To address the geometric characteristics of rail rails—long, thin, curved edges often accompanied by burrs—this invention introduces a dynamic serpentine convolution operator into the DS-ConvLSTM network. The parametric equations of the long edge profile in the local tangential-normal coordinate system are defined as follows: ; in The arc length parameter is along the rail direction. The orbital tangential unit vector. The unit vector is the normal vector on the outer side of the railhead. The amplitude of the fat edge protrusion. For local curvature. The dynamic serpentine convolution kernel in the first... The two-dimensional positions of each sampling point are: ; offset The gating network adaptively estimates based on local information from the feature map: ; The convolution output is a weighted sum along a serpentine path: ; To constrain the convolution kernel to closely follow the fat edge contour, a geometric energy term is introduced: ; The first term constrains the smooth distribution of sampling points, while the second term ensures that the sampling points are located in the edge regions with the largest gray-level gradients. , As weights. When training the DS-ConvLSTM network, As a regularization term added to the total loss, the convolution kernel is automatically deformed along the direction of the fat edge bulge, and the joint prediction result of the fat edge mask and three-dimensional dimensions (length, width, height) is output.

[0075] S3-4: BeiDou TCN-LSTM multi-task modeling and double-difference constraints.

[0076] High-frequency features of the BeiDou carrier phase sequence are extracted using TCN, and the dilation rate of the dilated convolutional layer is [1,2,4,8] to cover the BeiDou update interval; then, LSTM is used to capture the long-term temporal dependence of the orbit position. Let the first... The predicted position at each time point is The truth position is The clock difference is estimated as The multi-task loss is defined as follows: ; in For carrier phase adjacent epoch difference, For carrier frequency, This is the balance coefficient.

[0077] To further suppress the effects of ionospheric delay and satellite clock bias, the base station... and mobile station Carrier phase observations are processed using double-difference: ; in , For satellite Carrier phase observation, For reference satellite.

[0078] Double-difference observation residuals: ; in For carrier wavelength, This refers to the change in baseline geometric distance. The present invention will... As an auxiliary regularizer for TCN-LSTM, it significantly improves orbit positioning accuracy.

[0079] S3-5: Network training.

[0080] The DS-ConvLSTM network was trained using labeled fat-edge masks and 3D size data, and the network parameters were optimized using a combination of cross-entropy loss and size regression loss. The TCN-LSTM network was trained using the ground truth values ​​of position and clock difference, and optimized using the aforementioned multi-task loss. The network parameters were updated using Adam or SGD optimizers.

[0081] S3-6: Multi-sensor fusion (Kalman filtering).

[0082] The state vector is constructed using the fat edge spatial position measured by structured light, the orbital position output by TCN-LSTM, and the IMU output, and then fused using either Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF). The update formula is: ; in for State estimation after time-mapping For Kalman gain, For the observation vector, This is the observation matrix.

[0083] S3-7: Definition of the Spatiotemporal Consistency Index (STCI).

[0084] To provide a unified evaluation of the reliability of multi-sensor data in terms of morphology, location, and time, this invention proposes a spatiotemporal consistency index (STCI): ; in For the first The intersection-union ratio of the frame fat edge mask and the annotation mask. This represents the geographic coordinate error of the center of the fat edge in this frame. This represents the displacement residual along the track mileage direction. , The normalization coefficient is... The closer the STCI is to 1, the higher the morphological and spatial consistency of the detection in that frame.

[0085] S3-8: Adaptive Weighted Least Squares (AWLS) Optimization.

[0086] A globally robust estimation of the 3D size and spatial location of the fat edge after multi-sensor fusion is performed by introducing an adaptive weighted least squares objective function: ; in For the first The truth vector of the frame (including the length, width, height of the fat side and the center coordinates). This is the output of the system of the present invention. These are the parameters to be estimated. Weights. Determined by joint adaptive adjustment of STCI and residuals: ; in An index to enhance the influence of highly consistent samples, To prevent small constants with zero denominators, samples with high STCI and small residuals are weighted more heavily and play a dominant role in the overall estimation; outlier samples with low STCI or excessively large residuals are suppressed, thereby improving the robustness and accuracy of fat edge size and location estimation.

[0087] Preferably, the details of the construction method for the image display module in step 4 are as follows: S4-1: Point cloud-localization data fusion and coloring.

[0088] Based on the results of S3-1 and S3-6, the structured light point cloud and BeiDou-IMU positioning results are fused, and the rail point cloud is rendered in a unified orbit-geographic coordinate system. Normal rail areas are colored in grayscale (intensity). The fat border area is highlighted according to STCI, for example, the color is set to red (RGB=[1,0,0]), and the transparency is linearly related to STCI: ; The higher the STCI, the less transparent the fat-edge area, indicating a higher reliability of the test results.

[0089] S4-2: Location information annotation and error visualization.

[0090] Based on the Kalman filter posterior covariance matrix Calculate the parameters of the error ellipse on the horizontal plane, and draw the error ellipse for each fat edge center to represent the positioning uncertainty. Let... , for The eigenvalues ​​projected onto the horizontal plane, Given a chi-square distribution threshold at a confidence level (e.g., 95%), the major and minor semi-axes of the error ellipse are respectively... ; Furthermore, by overlaying 3D dimension text annotations, mileage markers, and geographic coordinates on the point cloud scene, the results of edge detection and the positioning error can be displayed intuitively.

[0091] The advantages of this invention lie in its integration of multiple sensors, including the BeiDou high-precision positioning system, a structured light camera, and an inertial measurement unit (IMU), to construct a unified track-geographic coordinate system. This allows for the precise recording and uploading of the geometric parameters of the rail's thick edge to the terminal. Through a PPS pulse and time synchronization algorithm, microsecond-level alignment between the BeiDou timestamp and the structured light and IMU data is achieved, significantly improving the accuracy of rail position information and spatial matching precision. Furthermore, this invention employs a structured light camera for non-contact measurement, avoiding potential secondary damage to the rail caused by traditional contact measuring tools. By projecting a structured light pattern and capturing its distortion on the rail surface, the structured light camera accurately reconstructs the spatial coordinates of various points in the rail head and thick edge area, constructing a high-resolution 3D model of the rail and providing a reliable data foundation for subsequent thick edge detail identification.

[0092] This invention constructs a DS-ConvLSTM network based on dynamic serpentine convolution. Its encoder uses the lightweight MobileNetV3 as the backbone and introduces a dynamic serpentine convolution layer to arrange sampling points in a serpentine pattern along the thick edge contour of the rail head. Through learnable offset and geometric energy constraints, it adaptively fits the slender, curved and burr-laden thick edge boundary. The decoder combines the ASPP (hollow spatial pyramid pooling) module to capture multi-scale contextual information and fuses it with the position state output by the Kalman filter. This effectively suppresses the influence of single-frame noise and local occlusion on the detection results, making the thick edge mask segmentation and 3D size estimation more stable and reliable. The positioning accuracy can be improved to the ±0.15mm level under typical working conditions.

[0093] In terms of navigation and spatial positioning, this invention introduces a TCN-LSTM network to perform multi-task time-series modeling of BeiDou carrier phase and differential observation sequences. The loss function simultaneously incorporates Euclidean distance and clock / phase difference constraints, and combines double-difference observations from a reference station to effectively suppress ionospheric delay, satellite clock error, and multipath effects. This reduces the spatial positioning accuracy of the track inspection vehicle from the traditional meter level to the centimeter level, providing a highly reliable geographic coordinate reference for accurate mileage marking and long-term tracking of edge defects.

[0094] In terms of multi-sensor data quality control and result optimization, this invention proposes a Spatiotemporal Consistency Index (STCI). This index weights and fuses information such as the intersection-union ratio (CUI) between the fat edge mask and the annotation, the geographic coordinate error of the fat edge center, and the mileage continuity along the route. It uniformly quantifies the reliability of each frame's detection result across the three dimensions of "morphology, spatial location, and time series." Based on this, an Adaptive Weighted Least Squares (AWLS) optimization framework is constructed. This framework uses STCI and residual magnitude to jointly determine sample weights, assigning greater weight to highly consistent samples and automatically reducing the weight of abnormal noise samples, thus achieving globally robust estimation of multi-sensor fusion results. Compared to methods using only simple weighted averaging or fixed-weight fusion, this invention significantly improves the stability and anti-interference capability of fat edge 3D size and spatial location estimation under complex working conditions.

[0095] In terms of results presentation and application, this invention constructs an integrated system for visualizing and positioning rail point clouds. It unifies the rendering of rail point clouds obtained from structured light reconstruction with BeiDou-IMU fused location information, highlighting and geometrically magnifying detected thick-edge areas. Simultaneously, it maps STCI to the transparency or color level of the thick-edge areas and overlays error ellipses generated by Kalman filtering posterior covariance, intuitively quantifying positioning uncertainty and significantly improving the reliability of expressing spatial information about rail defects. Employing an OpenGL 4.0 programmable pipeline combined with line-of-sight adaptive point cloud density control, it maintains a high frame rate while ensuring manageable GPU load, supporting real-time browsing and interactive analysis of large-scale rail point clouds and thick-edge detection results.

[0096] Furthermore, the overall architecture of this invention adopts a modular design. Key algorithms such as DS-ConvLSTM, TCN-LSTM, STCI, and AWLS can all have their parameters adjusted or network structures upgraded according to different line conditions and equipment configurations. It has good scalability and upgradeability, and can further optimize model performance as the data scale and computing power increase. It continuously improves the detection accuracy, positioning accuracy, and system robustness of rail edge defects, providing reliable technical support for railway line condition assessment and maintenance decisions.

[0097] Example 2 This embodiment provides a multimodal fusion dynamic serpentine convolution rail thick edge detection system, including: The data acquisition module is configured to acquire multi-source data required for rail edge detection, including rail 3D data, satellite positioning data, and inspection vehicle attitude information. The preliminary detection module is configured to: process the 3D data of the rail using a dynamic serpentine convolutional network, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region; The spatial positioning module is configured to: use a temporal convolutional network to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; The information fusion module is configured to: based on the inspection vehicle's attitude information, fuse the mask and 3D size estimation of the fat edge region with the spatial position information of the inspection vehicle, and calculate the spatiotemporal consistency index; The rail edge detection module is configured to use an adaptive weighted optimization algorithm, with the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail edge, and output the rail edge detection and positioning results.

[0098] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0099] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0100] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0101] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0102] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0103] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0104] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0105] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0106] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0107] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal fusion-based dynamic serpentine convolution method for detecting rail thick edges, characterized in that, Includes the following steps: To acquire multi-source data required for rail edge detection, the multi-source data includes three-dimensional rail data, satellite positioning data, and inspection vehicle attitude information; A dynamic serpentine convolutional network is used to process the 3D data of the rail, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region. Temporal convolutional networks are used to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; Based on the inspection vehicle's attitude information, the mask and 3D size estimation of the fat edge region are fused with the inspection vehicle's spatial position information, and the spatiotemporal consistency index is calculated. An adaptive weighted optimization algorithm is adopted, using the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail fat edge, and output the rail fat edge detection and positioning results.

2. The multimodal fusion dynamic serpentine convolution rail thick edge detection method as described in claim 1, characterized in that, The dynamic serpentine convolutional network is a DS-ConvLSTM network. Its convolutional kernel sampling points can be adaptively distributed in a serpentine pattern according to the slender curved profile of the rail fat side. It is optimized by learnable offset and geometric energy constraints so that the convolutional kernel closely follows the boundary of the fat side.

3. The multimodal fusion dynamic serpentine convolution rail thick edge detection method as described in claim 1, characterized in that, The temporal convolutional network is a TCN-LSTM network, which extracts local and high-frequency features of satellite positioning data through the TCN network and models the long-term temporal dependence of rail position through the LSTM network. The loss function used during network training includes both position error and clock / phase difference constraints.

4. The multimodal fusion dynamic serpentine convolution rail thick edge detection method as described in claim 1, characterized in that, The multi-source data is synchronized through hardware triggering and unified timestamps. Specifically, the PPS pulse signal output by the satellite positioning receiver is used to trigger the exposure of the structured light camera and the sampling of the IMU, and the time synchronization algorithm is used to achieve microsecond-level time alignment.

5. The multimodal fusion dynamic serpentine convolution rail thick edge detection method as described in claim 1, characterized in that, The spatiotemporal consistency index is calculated using the following formula: ； in, For the first The cross-union ratio between the frame fat edge mask and the annotation mask. This represents the geographic coordinate error of the center of the fat edge in this frame. This represents the displacement residual along the track mileage direction. , The normalization coefficient is... λ 1, λ 2, λ 3 represents the weighting coefficient.

6. The multimodal fusion dynamic serpentine convolution rail thick edge detection method as described in claim 1, characterized in that, The adaptive weighted optimization algorithm is an adaptive weighted least squares method, and its objective function is: ； in, For the first The truth vector of a frame. For system output, For the parameter to be estimated, the weights are... Determined by joint adaptive analysis of spatiotemporal consistency index and residuals: ； in, An index to enhance the influence of highly consistent samples, To prevent small constants with a denominator of zero.

7. A multimodal fusion dynamic serpentine convolution rail thick edge detection system, characterized in that, include: The data acquisition module is configured to acquire multi-source data required for rail edge detection, including three-dimensional rail data, satellite positioning data, and inspection vehicle attitude information. The preliminary detection module is configured to: process the 3D data of the rail using a dynamic serpentine convolutional network, extract the contour features of the thick edge of the rail, and output the mask and 3D size estimate of the thick edge region; The spatial positioning module is configured to: use a temporal convolutional network to perform temporal modeling on satellite positioning data to obtain the spatial location information of the inspection vehicle; The information fusion module is configured to: based on the inspection vehicle's attitude information, fuse the mask and 3D size estimation of the fat edge region with the spatial position information of the inspection vehicle, and calculate the spatiotemporal consistency index; The rail edge detection module is configured to use an adaptive weighted optimization algorithm, with the spatiotemporal consistency index as the weight basis, to globally optimize the three-dimensional dimensions and spatial position of the fused rail edge, and output the rail edge detection and positioning results.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images