Rail corrugation detection method and system based on multi-modal feature fusion

Abstract

The application discloses a rail corrugation detection method and system based on multi-modal feature fusion, relates to the technical field of track engineering intelligent operation and maintenance, and comprises the following steps: collecting axle box vibration acceleration, wheel-rail noise and high-definition video signals of a rail surface area in a train operation process, and marking a mileage position in combination with an optical-electricity odometer; identifying a pixel distance length of a rail surface wave-shaped wear wavelength feature through a deep learning model, and performing spatial transformation to obtain a rail corrugation wavelength value in combination with camera calibration parameters; performing second-level analysis and processing on the real-time acquired axle box vibration acceleration signals and wheel-rail rolling noise signals by using a narrowband spectrum, and inversely calculating a rail corrugation wave depth value at a current position based on a neural network model; and determining key modal parameters of the rail corrugation according to the wavelength and the wave depth value of the rail corrugation, and performing quantitative evaluation on rail corrugation diseases. The method can complete early identification, dynamic evaluation and active early warning of the rail corrugation, and reduce rail maintenance and repair costs.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for rail engineering, and in particular to a rail corrugation detection method and system based on multimodal feature fusion. Background Technology

[0002] Rail corrugation (hereinafter referred to as "rail corrugation") is one of the main excitation sources that commonly induce strong vibrations and rolling noise in vehicle-track systems. Specifically, it manifests as periodic wavy wear on the top surface of the rail along the longitudinal direction. Essentially, it is the plastic flow and wear generated during the rolling contact between the wheel and rail. Once rail corrugation forms, it becomes a strong periodic excitation source, causing severe vibrations and impacts in the wheel-rail system, leading to low-frequency structural noise in underground lines or wheel-rail whistling noise in elevated lines.

[0003] Currently widely used rail corrugation detection technologies rely on manual labor, resulting in high rates of missed and false detections. Furthermore, because detection operations are limited by normal train operating hours, they are typically conducted during scheduled track maintenance windows, leading to delays in defect detection. Moreover, the cost of maintenance work based on production plans is higher than timely repairs in the early stages of defect development. In my country's rail transit industry, the cost of rail replacement due to severe rail corrugation defects accounts for a significant portion of rail damage repair costs annually. The structured light vision inspection method, which has emerged in recent years, is constrained by complex operating conditions and tunnel environments, making it difficult to extend to online continuous inspection and still unable to overcome the limitations of track maintenance windows. Existing online rail corrugation detection technologies typically use vibration signals as an indirect detection method; this reliance on a single data type usually limits the detection dimensions and accuracy. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a rail corrugation detection method based on multimodal feature fusion to solve the problems of early identification, dynamic tracking and accurate early warning of rail corrugation in existing technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a rail corrugation detection method based on multimodal feature fusion, which includes acquiring high-definition video signals of axle box vibration acceleration, wheel-rail noise and rail surface area during train operation, and marking mileage positions in combination with photoelectric odometer; Dynamic detection is performed on the real-time acquired video signal. The pixel distance length of the wave-shaped wear wavelength characteristics of the rail surface is identified through a deep learning model. The spatial transformation is combined with the camera calibration parameters to obtain the rail wave wavelength value. The narrowband spectrum is used to perform second-level analysis and processing on the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal. The frequency domain feature parameters of vibration and noise are used as real-time inputs, and the rail corrugation depth value at the current position is inverted based on the pre-trained neural network model. Based on the wavelength and depth of rail corrugation, the key morphological parameters of rail corrugation are determined, and a comprehensive comparison with track maintenance rules is conducted to quantitatively evaluate rail corrugation defects.

[0007] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, the acquisition of high-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area during train operation includes using a single bogie as a measurement unit, and the layout of measurement points in each measurement unit includes: The axle box vibration acceleration is measured at the center of the axle box cover surface of the left and right wheels, with one vertical measurement direction set on each side; the wheel-rail noise is measured near the wheels on both sides of the bogie, with one acoustic measurement point set on each side, and the distance from the wheel-rail contact point is less than the maximum measurement distance. High-definition video of the track surface area is captured from the bottom of the bogie, directly facing the track surface area, with one video monitoring point set on each side; the photoelectric odometer is set at the end of the non-drive wheel axle of the bogie, with a total of one measuring point, and the measured value is regarded as the travel distance of the whole vehicle.

[0008] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, the pixel distance length for identifying the wave-shaped wear wavelength features of the rail surface through a deep learning model includes: before the detection begins, setting up an image acquisition system and calibrating the camera; and acquiring high-definition videos of the left and right rail surfaces in real time during train operation. Target detection is performed using a pre-trained deep learning model, and the recognition unit is the bounding rectangle of a single valley edge contour region. The pixel coordinates of the centers of adjacent detection frames for rail corrugation detected by the deep learning model are combined with the camera calibration results before each detection to perform spatial coordinate transformation and obtain the true distance, which is the wavelength value. .

[0009] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, wherein: the step of obtaining the true wavelength value by combining camera calibration parameters to perform spatial transformation includes establishing a vertically shooting visual inspection system: a high-performance industrial camera is located directly above the rail, the lens axis of the camera is perpendicular to the rail surface, and the camera focuses and shoots vertically downwards; The camera calibration adopts the Zhang Zhengyou calibration method to obtain the camera's internal and external parameters, including focal length, principal point coordinates, rotation matrix, and translation vector, and to establish the mapping relationship between the pixel coordinate system and the world coordinate system. Based on the dataset constructed from high-resolution images of rail corrugation and labeled rail corrugation detection boxes, a target detection model for rail corrugation recognition is obtained through deep learning training; both the labeled detection boxes and the detection boxes output during actual operation are bounded rectangles of the contour region of each corrugation trough. The pixel coordinate results of the centers of adjacent detection boxes for rail corrugation returned by the deep learning model specifically include: During online detection, a trained model is used for real-time inference, outputting the detected data. A set of valley bounding boxes is generated, and the center pixel coordinates of each valley bounding box are calculated. All center points are sorted according to their horizontal coordinates to obtain an ordered center point sequence. Based on the ordered center point sequence, the pixel distance between adjacent valleys and the average wavelength are calculated. After the average wavelength is calculated, the detection quality of the image frame is evaluated. Combining the camera calibration results before each detection, and based on the deep learning model, the pixel coordinates of the centers of adjacent detection boxes of rail corrugations are detected and mapped to convert the pixel length into the actual rail corrugation wavelength. By detecting each continuous image frame extracted from the online video stream or video, the time series of rail corrugation wavelengths during train travel is obtained, which is the information sequence of rail corrugation wavelength and track mileage.

[0010] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, the step of performing second-level analysis and processing on the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal using narrowband spectrum includes: performing signal processing on the axle box vibration acceleration signal and wheel-rail rolling noise signal within the current moment and adaptive window length range using narrowband spectrum; and setting an adaptive window length for the current moment. For the measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Frequency domain processing is performed, using narrowband spectrum as a signal processing method; For a single narrowband signal, since the cosine function couples information to two sidebands, the analytic signal of the real signal is defined as a complex signal: The spectrum of a complex signal is concentrated near the carrier frequency. By frequency shifting to the baseband, the complex envelope of the signal can be obtained. ; Obtain complex envelope Represented in rectangular coordinates, it is denoted in the frequency domain as a low-pass spectrum concentrated near zero frequency. ; The measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Each signal is fully encoded into a low-pass signal, containing all the modulation information of the original high-frequency narrowband signal.

[0011] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, wherein: the rail corrugation depth value at the current position is inverted based on the pre-trained neural network model, which includes taking the frequency domain feature parameters of vibration and noise within the window length range as real-time input; The frequency domain characteristic parameters after narrowband spectrum processing include three indicators: centroid frequency, root mean square frequency, and spectral entropy. Among them, centroid frequency and root mean square frequency describe the location distribution of the main frequency band, while spectral entropy describes the uniformity and randomness of the frequency component distribution. Measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power After processing the narrowband spectrum, the frequency domain signals are obtained respectively. Time series signal of wheel-rail rolling noise sound power The three indices of centroid frequency, root mean square frequency, and spectral entropy were extracted to obtain the following set of input parameters for the neural network. The system rapidly infers the rail corrugation depth at the current location using a pre-trained neural network model, specifically including: Based on a pre-trained neural network model, the input parameter set and the rail corrugation wavelength Output the corrugation depth of the rail at the corresponding position. The prediction results constitute the complete rail corrugation shape characteristics at the current moment and current mileage position; The pre-trained neural network model is based on a backpropagation neural network and consists of an input layer, several hidden layers, and an output layer. The nodes between the layers are connected by weights. The connection weights are iteratively adjusted using a supervised learning algorithm based on the principle of error backpropagation. The training data is based on previously measured axle box vibration acceleration and wheel-rail rolling noise, as well as measured rail corrugation wavelength and depth at corresponding locations. During model training, the learning process of the backpropagation neural network includes two stages: forward propagation and weight update. In the forward propagation stage, the input data is processed sequentially through each layer of the network and converted into output through a nonlinear function. After the output is generated, it is compared with the target value to calculate the error. The error is then propagated back to the network in a reverse manner. The chain rule is used to effectively calculate the gradient of the loss function with respect to each weight, and the adjustment responsibility of each weight is quantified. During the weight update phase, gradient descent is used to adjust the weights; the process is iterated to obtain a pre-trained neural network model, which is then used for real-time inference during the online detection process.

[0012] As a preferred embodiment of the rail corrugation detection method based on multimodal feature fusion described in this invention, the quantitative evaluation of rail corrugation defects includes: fusing data association, using the mileage position marked by the photoelectric odometer as a reference, binding the output rail corrugation wavelength value and rail corrugation depth value with the mileage position, constructing a three-dimensional data matrix of mileage-depth-wavelength, and performing data alignment. Based on the rail corrugation wavelength value at the current time or location, and according to the wavelength classification standard, determine whether the defect type at the current time or location is long-wavelength corrugation or short-wavelength corrugation. Based on the current rail corrugation depth value at the current time or location, and combined with the determined wavelength properties, if the limit is exceeded, the rail damage is considered to have reached the level of serious damage. Based on the integrated analysis of vibration, noise, and visual data, the system outputs the detection results and judgment conclusions of rail corrugation at the current time or location.

[0013] Secondly, this invention provides a rail corrugation detection system based on multimodal feature fusion, comprising: a first part, a hardware architecture integrating a multimodal edge sensing module to construct an on-board detection network; the main components include: Vibration and noise integrated sensing data acquisition module: Deployed in key locations of the train running gear, it is equipped with a multi-channel data acquisition system, a large-range piezoelectric accelerometer, an acoustic microphone, and a data connection cable; Through the accelerometer and noise sensor, it collects vibration acceleration data of the bogie axle box and noise data of the running gear in real time during normal train operation; Visual perception and video image acquisition module: Deployed on the frame structure near the wheel-rail contact area of ​​the train, it is equipped with an industrial camera and auxiliary light source; it is responsible for acquiring visual signals and capturing high-definition video or images at a high frame rate during train operation to ensure continuous sampling of the rail surface; The acquisition module is based on an on-board multimodal edge perception fusion detection unit. It utilizes predefined key measurement points of the train running gear and pre-set data acquisition devices to achieve synchronous acquisition and preliminary processing of axle box vibration, wheel-rail noise, rail surface images and mileage information. The second part is the analysis architecture and diagnostic equipment of the online detection system, which is responsible for the deep fusion and intelligent analysis of multimodal data collected by edge hardware, integrating multimodal spatiotemporal feature fusion, and performing fusion analysis of visual and vibration noise to achieve automatic identification and dynamic evaluation of rail corrugation.

[0014] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the rail corrugation detection method based on multimodal feature fusion as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the rail corrugation detection method based on multimodal feature fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: The online detection method overcomes the limitations of traditional rail corrugation detection methods in terms of operating time. Installing this invention on operating trains enables all-weather online detection, significantly improving operational efficiency and coverage. Relying on vehicle-mounted deployment and data collection during normal train operation, it overcomes the limitations of traditional manual inspections, such as nighttime operation windows, cumbersome procedures, slow detection speed, and limited single-time coverage. Efficiency is several times higher than traditional manual operations, pushing traditional static detection towards rapid, high-frequency, full-line dynamic detection. A multi-modal edge-sensing fusion detection system is established, deeply integrating vibration, noise, and visual sensing technologies at the edge. Visual technology accurately measures the wavelength value of rail corrugation, while non-contact sensing of the wave depth value of rail corrugation via vehicle-mounted vibration and noise signals achieves coordinated and precise detection of mileage, wavelength, and wave depth. This overcomes the limitations of traditional vehicle-mounted detection technologies, such as inaccurate wave depth, blurred wavelength, and positional deviation due to single-modal detection, enabling full-element dynamic detection of rail corrugation defects. This AI-driven rail corrugation feature analysis, relying on a lightweight neural network model, performs real-time fusion analysis of multi-source data. It supports feature recognition and dynamic evaluation of rail corrugation, possessing self-learning iteration and continuous optimization capabilities. Through continuously accumulating measured data, it optimizes the neural network and visual recognition model, constantly improving detection accuracy and diagnostic precision. Compared to traditional manual inspection trolleys, this invention is lower in cost, more universal, supports rapid retrofitting of existing trains, and features small, easy-to-assemble and disassemble, and high-safety components. It can be implemented in batches on urban rail networks, enabling multi-train data linkage analysis to generate a line-level corrugation defect distribution heat map, providing data support for overall line maintenance planning and achieving a full-network coverage, 24 / 7 online status early warning and maintenance mode. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a rail corrugation detection method based on multimodal feature fusion.

[0019] Figure 2This is a topology diagram of a rail corrugation detection method based on multimodal feature fusion.

[0020] Figure 3 This diagram illustrates the multimodal data analysis method for rail corrugation detection based on multimodal feature fusion.

[0021] Figure 4 This diagram illustrates the signal processing method for rail corrugation detection based on multimodal feature fusion.

[0022] Figure 5 This is a schematic diagram of the rail corrugation detection indexes for a rail corrugation detection method based on multimodal feature fusion.

[0023] Figure 6 This is a cloud platform for intelligent diagnosis of rail corrugation in subway lines, based on a multimodal feature fusion method for rail corrugation detection.

[0024] Figure 7 This is a schematic diagram of the original keyframe image for extracting the wavelength features of rail corrugation in a rail corrugation detection method based on multimodal feature fusion.

[0025] Figure 8 This is a schematic diagram of the detection results of keyframe images for rail corrugation wavelength feature extraction based on the rail corrugation detection method using multimodal feature fusion.

[0026] Figure 9 This is a schematic diagram showing the vertical vibration acceleration results of the axle box under the condition of comparing the presence and absence of rail corrugation in the rail corrugation detection method based on multimodal feature fusion.

[0027] Figure 10 This is a schematic diagram showing the time-domain characteristic parameters of the vertical vibration acceleration of the axle box under different wave depth comparison conditions of the rail corrugation detection method based on multimodal feature fusion.

[0028] Figure 11 This is a schematic diagram showing the sound power level of wheel-rail rolling noise under specific corrugation wavelength and different amplitude wave depth comparison conditions for the rail corrugation detection method based on multimodal feature fusion. Detailed Implementation

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0031] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0032] Reference Figures 1-5 This is one embodiment of the present invention, which provides a rail corrugation detection method based on multimodal feature fusion, comprising the following steps: S1: Collects high-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area during train operation, and marks the mileage position using a photoelectric odometer.

[0033] The acquisition of high-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area during train operation includes using a single bogie as a measurement unit, with the measurement points of each unit arranged as follows: The axle box vibration acceleration is measured at the center of the axle box cover surface of the left and right wheels, with one vertical measurement direction set on each side; the wheel-rail noise is measured near the wheels on both sides of the bogie, with one acoustic measurement point set on each side, and the distance from the wheel-rail contact point is less than the maximum measurement distance.

[0034] High-definition video of the track surface area is captured from the bottom of the bogie, directly facing the track surface area, with one video monitoring point set on each side; the photoelectric odometer is set at the end of the non-drive wheel axle of the bogie, with a total of one measuring point, and the measured value is regarded as the travel distance of the whole vehicle.

[0035] A smart online detection method for rail corrugation based on vibration-noise-visual multimodal feature fusion was used to synchronously acquire multimodal data. The predefined real-time multimodal data measurement used a single bogie as the measurement unit. The specific measurement points for each unit were as follows: axle box vibration acceleration was measured at the center of the axle box cover surface of the left and right wheels, with one vertical measurement direction on each side; wheel-rail noise was measured near the wheels on both sides of the bogie, with one acoustic measurement point on each side, and the distance from the wheel-rail contact point should not exceed 350mm; high-definition video of the rail surface area was captured from the bottom of the bogie, directly facing the rail surface area, with one video monitoring measurement point on each side; and a photoelectric odometer was installed at the end of the non-drive wheel axle of the bogie, with one measurement point, and the measured value was considered as the total travel distance of the entire vehicle.

[0036] In the S1 multimodal data synchronous acquisition of the above-mentioned intelligent online detection method for rail corrugation based on the fusion of vibration-noise-visual multimodal features, the axle box vibration acceleration is the vibration acceleration signal of the axle box of the train running gear in the vertical direction (i.e., perpendicular to the rail surface), reflecting the intensity of wheel-rail dynamic interaction, and is the core correlation parameter of the wave depth feature, with a sampling frequency of not less than 5000Hz; the main components of wheel-rail noise include wheel-rail contact friction, airborne sound signals generated by impact, etc., which are directly related to the periodic characteristics of corrugation, and assist the vibration signal to improve the accuracy of wave depth calculation, with a sampling frequency of not less than 20000Hz; high-definition video of the rail surface is used to extract the spatial periodic characteristics (i.e., wavelength) of rail surface corrugation, and the video frame corresponds to the rail surface position determined by the photoelectric odometer, with a sampling frequency of not less than 30FPS; the mileage position mark records the number of wheel rotations through the photoelectric odometer, which is converted into the train running mileage, realizing the accurate binding of each frame of data with the line mileage, and the mileage positioning error is not higher than 0.1m.

[0037] In the above-mentioned intelligent online detection method for rail corrugation based on the fusion of vibration-noise-visual multimodal features, the pre-set data acquisition and processing equipment includes dynamic data acquisition equipment, large-range piezoelectric accelerometer, industrial-grade microphone, industrial camera, photoelectric odometer, edge computing processor, etc.

[0038] Lightweight sensing equipment is installed on operating vehicles to enable 24 / 7 online monitoring of rail corrugation during train operation. Data sources primarily include vibration signals, noise signals, and track surface images. The data acquisition system comprises dynamic data acquisition equipment, a large-range piezoelectric accelerometer, industrial-grade microphones, industrial cameras, photoelectric odometers, and edge computing processors. The data acquisition system supports multi-channel asynchronous sampling and, in conjunction with the edge computing module, supports dual-layer encrypted data storage and wireless transmission.

[0039] The vibration signal originates from the axle box acceleration. The selection of measurement points is based on the fact that rail corrugation primarily involves unevenness on the rail surface in the vertical direction, thus significantly impacting the vertical vibration acceleration of the axle box. At the center of the axle box cover corresponding to the selected wheel position, one accelerometer is installed on each side of the axle box cover surface. The contact surface should be flat and smooth. After being fixed with tooling and structural adhesive, the accelerometer should be positioned in a vertical plane to measure the axle box acceleration in the vertical direction on one side. The piezoelectric accelerometer should have factory sensitivity calibration, and the sensitivity value should be known. The recommended sensor range is 100g, ensuring no signal loss or sensor failure due to current overload when a train load passes.

[0040] The noise signal originates from wheel-rail rolling noise. Wheel-rail interaction not only causes vibration responses in the vehicle and track systems but also generates strong wheel-rail noise. In urban rail transit, rolling noise is the most significant source of wheel-rail noise. Periodic wear on the wheel-rail surface causes vertical vibration of the wheel-rail system, and the wavelength of wheel-rail roughness related to rolling noise is mainly in the range of 5mm to 500mm. Therefore, the presence of rail corrugation excites vibrations in specific frequency bands of the wheel-rail system, which then transmits noise to the track and wheel structures, thus radiating it. Therefore, one noise sensor is placed near each wheel on each side corresponding to the selected wheel position. The recommended parameters for the sensors are as follows: frequency response 20~20000Hz, sensitivity 50±5mV / Pa, dynamic range 30~130dB(A), sampling frequency ≥4000Hz.

[0041] The video signal comes from high-definition video of the rail surface area captured during train operation, obtained by an industrial camera. The camera is fixedly placed in a groove at the bottom of the train bogie or locked to a custom fixture and bonded to the bottom of the bogie. The shooting angle should be vertical, i.e., the lens should be pointed at the rail surface area. The frame rate and resolution are set, and a high frame rate industrial camera should be used to ensure an appropriate time step for calculation if frame-by-frame fine analysis is required. Real-time motion images are transmitted to the edge computing end, and the computer program calls a pre-trained model file to perform rail corrugation detection. Before detection begins, the camera's internal and external parameters should be obtained through camera calibration to convert pixel distances to the actual rail corrugation wavelength during calculation. The pre-trained model is trained using the methods described in step S2.

[0042] Figure 2 This is a topology diagram of an intelligent online detection method for rail corrugation based on the fusion of vibration-noise-visual multimodal features. It includes the overall technical framework of perception layer-data layer-output, and embeds key technologies such as data layer fusion and feature layer fusion. Figure 2 This section describes the multimodal data acquired by each hardware device in step S1, and the subsequent signal processing flow. This mainly includes detecting the wavelength characteristics of rail corrugation using a lightweight convolutional neural network model (corresponding to step S2) and using the acquired output wavelength as key prior knowledge to predict the wave depth characteristics of rail corrugation using a lightweight neural network model (corresponding to step S3). A detailed flowchart of the signal processing technology can be found in [link to flowchart]. Figure 3 .

[0043] S2: Dynamically detect the real-time acquired video signal, identify the pixel distance length of the wave-shaped wear wavelength characteristics of the rail surface through a deep learning model, and obtain the rail wave wavelength value by spatial transformation in combination with camera calibration parameters.

[0044] The pixel distance length for identifying the wave-shaped wear wavelength features of the rail surface using a deep learning model includes: before detection begins, setting up an image acquisition system and calibrating the camera; and acquiring high-definition videos of the left and right rail surfaces in real time during train operation.

[0045] Target detection is performed using a pre-trained deep learning model, and the recognition unit is the bounding rectangle of a single valley edge contour region.

[0046] Based on the pixel coordinates of the centers of adjacent detection frames for rail corrugation detected by the deep learning model, and combined with the camera calibration results before each detection, a spatial coordinate transformation is performed to obtain the true distance, which is denoted as the wavelength value. .

[0047] The process of obtaining the true wavelength value by combining camera calibration parameters with spatial transformation includes establishing a vertically shooting visual inspection system: a high-performance industrial camera is located directly above the rail, with the camera's lens axis perpendicular to the rail surface, focusing and shooting vertically downwards.

[0048] The camera calibration adopts the Zhang Zhengyou calibration method to obtain the camera's internal and external parameters, including focal length, principal point coordinates, rotation matrix, and translation vector, and to establish the mapping relationship between the pixel coordinate system and the world coordinate system.

[0049] Based on the dataset constructed from high-resolution images of rail corrugation and labeled rail corrugation detection boxes, a target detection model for rail corrugation recognition is obtained through deep learning training. Both the labeled detection boxes and the detection boxes output during actual operation are bounded rectangles of the contour region of each corrugation trough.

[0050] The pixel coordinate results of the centers of adjacent detection boxes for rail corrugation returned by the deep learning model specifically include: During online detection, a trained model is used for real-time inference, outputting the detected data. A set of valley bounding boxes is generated. For each valley bounding box, the center point pixel coordinates are calculated. All center points are sorted according to their horizontal coordinates to obtain an ordered center point sequence. Based on the ordered center point sequence, the pixel distance between adjacent valleys and the average wavelength are calculated. After the average wavelength is calculated, the detection quality of the image frame is evaluated.

[0051] Combining the camera calibration results before each detection, and based on the deep learning model, the pixel coordinates of the centers of adjacent detection boxes of rail corrugations are detected and mapped to convert the pixel length into the actual rail corrugation wavelength. By detecting each continuous image frame extracted from the online video stream or video, the time series of rail corrugation wavelengths during train travel is obtained, which is the information sequence of rail corrugation wavelength and track mileage.

[0052] In the S2 wavelength feature extraction of the above-mentioned intelligent online detection method for rail corrugation based on vibration-noise-visual multimodal feature fusion, the main working principle of this step is to capture high-definition images of the rail surface using an industrial camera, transmit them to a computer via a wireless router, and perform operations such as key area extraction, coordinate system transformation, tilt correction, and corrugation localization to finally obtain the wavelength of the rail corrugation. The mileage location corresponding to the occurrence of rail corrugation is output by the photoelectric odometer as the axle rotates. The visual perception module is mainly used to acquire the wavelength information of rail corrugation and accurately locate key sections, providing precise spatial location and quantitative indicators for a comprehensive assessment of the development of rail corrugation. The main technical approaches include: S21. Before the detection begins, set up the image acquisition system and calibrate the camera.

[0053] S22. High-definition video of the left and right sides of the track surface is acquired in real time during train operation.

[0054] S23. Target detection is performed using a pre-trained deep learning model, and the recognition unit is the bounding rectangle of a single valley edge contour region.

[0055] S24. Based on the pixel coordinates of the centers of adjacent detection frames of rail corrugation detected by the deep learning model, and combined with the camera calibration results before each detection, a spatial coordinate transformation is performed to obtain the true distance, which is the wavelength value. .

[0056] The image acquisition system mentioned in step S21 above specifically refers to establishing a vertically focused visual inspection system: a high-performance industrial camera is positioned directly above the rail, with the camera's lens axis perpendicular to the rail surface, focusing and capturing images vertically downwards. The installation height is... Should meet: in, It is the width of the top surface of the rail. Common values ​​for rail models include: 70mm for 50kg / m rails, 73mm for 60kg / m rails, and 75mm for 75kg / m rails. This is the horizontal field of view of the camera, and its specific value is determined based on the camera's performance parameters. The camera calibration mentioned in step S21 above, performed before each detection, uses the Zhang Zhengyou calibration method to obtain the camera's internal parameters (including focal length, principal point coordinates, etc.) and external parameters (including rotation matrix, translation vector, etc.), establishing the mapping relationship between the pixel coordinate system and the world coordinate system. Its mathematical expression can be written as: In this context, any point in the world coordinate system is defined as... Any point in the image pixel coordinate system is defined as , As a scale factor, Let be the rotation matrix of the camera's extrinsic parameters. Let be the translation vector of the camera's extrinsic parameters, and let be the camera's intrinsic parameter matrix. The mathematical expression is: The focal lengths of the camera's two axes are defined as follows: and The principal point coordinates are Combining the optimized Zhang Zhengyou calibration method, a checkerboard pattern is used for camera calibration. Known lengths can be directly measured on the checkerboard pattern. The length of the corresponding pixel measured by computer vision technology is defined as Then the camera scaling factor for the corresponding test should be... for: This coefficient will be used directly to convert the pixel length into the actual rail corrugation wavelength after the detection is completed.

[0057] In step S22 above, the industrial camera's shooting mode during train operation is continuous shooting or video frame capture; the specific lighting conditions are auxiliary lighting conditions from the onboard LED light source, with side supplementary lighting designed to ensure uniform image illumination without shadows or rail surface reflections, and the image width direction parallel to the longitudinal edge of the rail surface. The pre-trained deep learning model mentioned in step S23 above, based on the Python language and PyTorch framework, utilizes the preferred conventional YOLO algorithm and base model. Based on a dataset constructed from a large number of high-definition images of rail corrugation and manually labeled rail corrugation detection boxes, a target detection model capable of recognizing rail corrugation is obtained through deep learning training. Both the manually labeled detection boxes and the detection boxes output during actual operation are bounded rectangles of the contour region of each corrugation trough, with the pixel coordinates of their diagonal points written as: The above is Detection frame, and These are the coordinates of the top left and bottom right corners of the detection box, respectively. In step S24 above, the pixel coordinates of the centers of adjacent detection boxes for rail corrugation detected by the deep learning model specifically refer to: during online detection, the model trained in step S23 is used for real-time inference, and the direct output of the model is the detected... A set of trough bounding boxes For each detection box The center pixel coordinates need to be calculated separately. Its mathematical expression is written as: Sort all center points according to their horizontal coordinates to obtain an ordered sequence. : Calculate the pixel distance between adjacent valleys based on an ordered sequence of center points. : Methods for calculating average wavelength: in, This represents the average value of the rail corrugation wavelength at the pixel distance scale. This represents the number of valley boundary detection boxes directly output by the model. After the average wavelength is calculated, the detection quality of that image frame should be evaluated. Define the standard deviation. The calculation method for using the single-frame image detection result as a quality evaluation index is as follows: in, For the first Frame image detection quality evaluation metrics For the first The first frame of the image The pixel distance between each valley boundary detection box and its adjacent valley boundary detection boxes. The detection result is considered reliable if this metric meets the following condition: Conversely, if the result is not found to be reliable, the test result is considered unreliable, and the method used is... -1 and The detection result of frame +1 is linearly interpolated to obtain the detection result of frame j. In step S24 above, the true distance is obtained by combining the camera calibration results before each detection. Specifically, this means: combining the camera calibration results before each detection, and based on the pixel coordinates of the centers of adjacent detection boxes of rail corrugation returned by the deep learning model, coordinate mapping is performed to convert the pixel length into the true rail corrugation wavelength. The calculated rail corrugation wavelength value at this time The corresponding confidence level for: in, The calibration error is obtained in step S21 according to the preferred Zhang Zhengyou calibration method. As the rail corrugation wavelength value detected in the image frame, according to the sub-steps S21~S24 involved in step S2, the continuous image frames of the online video stream or video extraction can be detected one by one to obtain the time series of rail corrugation wavelength during the train's movement, that is, the information sequence of rail corrugation wavelength and line mileage.

[0058] Image grayscale conversion. Video captured by vehicle-mounted industrial cameras can be extracted into high-definition color RGB images. Grayscale processing refers to assigning the same values ​​to the red, green, and blue primary color channels of an image, meaning that each pixel in the pixel matrix has equal values ​​for all three channels. This reduces the workload of subsequent processing, lowers the difficulty of model training, and improves recognition speed. Grayscale calculation methods are generally divided into component methods, semi-mean methods, maximum value methods, and weighted average methods. This paper uses the weighted average method for calculation, and the calculation method is as follows: Where gray represents the grayscale component. The red component, For the green component, The blue component.

[0059] Gaussian filtering for images. Gaussian filters suppress noise and smooth out half-images, and compared to mean filters, they have the advantage of less blurring. Gaussian filtering is often implemented using discrete window convolution, and the calculation method is as follows: in, Given the Gaussian filter window length, the Gaussian kernel function is calculated using the following formula: in, is the standard deviation of the Gaussian distribution.

[0060] Histogram equalization. While filtering suppresses noise, it also causes some loss of image detail, necessitating image enhancement. The enhancement method used is histogram equalization, which calculates pixel normalization probabilities using a transformation function to obtain the grayscale values ​​of each pixel after histogram equalization. Histogram equalization transforms the image's grayscale values ​​into a uniformly distributed form across the entire grayscale range and enhances image contrast by adjusting the dynamic range of grayscale values. Comparing the grayscale processing results reveals that the target object in the image is more prominent. Ideally, each grayscale value corresponds to the same number of pixels. The pixel value calculation method for adaptive histogram equalization is as follows: in, These are the pixel values ​​before and after the transformation. H is the average grayscale value of the pixels, where H is the average grayscale value of the pixels. The transformation function k is the weighting coefficient.

[0061] Distortion correction: Using the camera's calibrated internal parameters and distortion coefficients, the image is corrected to eliminate radial and tangential distortion of the lens, thus ensuring geometric accuracy.

[0062] Specified region stitching: Utilizing the scale invariance and rotation invariance of the SIFT (Scale Invariant Feature Transform) algorithm, key feature points are detected and matched in adjacent frames. The Kd-tree (K-dimensional tree) algorithm is used for feature matching and the RANSAC algorithm is used to filter feature points and reduce matching errors. Finally, image stitching is completed, and image fusion is achieved using a weighted average method to eliminate stitching artifacts.

[0063] Training and Evaluation: By manually labeling the wave pattern, we construct training, validation, and test sets, prepare the training catalog, select the best YOLO pre-trained models, determine the model training hyperparameters, complete the training, evaluate the model using cross-validation, and check the loss value, accuracy, recall, and other metrics in the training logs.

[0064] Through the above steps, a pre-trained model is obtained, which can be directly used for real-time detection of video streams. The output pixel coordinates are combined with camera calibration parameters to achieve transformation and calculate the rail corrugation wavelength value.

[0065] S3: The narrowband spectrum is used to perform second-level analysis and processing on the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal. The frequency domain feature parameters of vibration and noise are used as real-time inputs, and the rail corrugation depth value at the current position is inverted based on the pre-trained neural network model.

[0066] The second-level analysis and processing of the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal using narrowband spectrum includes: performing signal processing on the axle box vibration acceleration signal and wheel-rail rolling noise signal within the current moment and adaptive window length range using narrowband spectrum; and setting the adaptive window length for the current moment.

[0067] For the measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Frequency domain processing is performed.

[0068] For a single narrowband signal, since the cosine function couples information to two sidebands, the analytic signal of the real signal is defined as a complex signal: The spectrum of a complex signal is concentrated near the carrier frequency. By frequency shifting to the baseband, the complex envelope of the signal can be obtained. .

[0069] Obtain complex envelope Represented in rectangular coordinates, it is denoted in the frequency domain as a low-pass spectrum concentrated near zero frequency. .

[0070] Measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Each signal is fully encoded into a low-pass signal, containing all the modulation information of the original high-frequency narrowband signal.

[0071] The method for retrieving the rail corrugation depth value at the current location using a pre-trained neural network model involves using frequency domain feature parameters of vibration and noise within a window length range as real-time input.

[0072] The frequency domain characteristic parameters after narrowband spectrum processing include three indicators: centroid frequency, root mean square frequency, and spectral entropy. Among them, centroid frequency and root mean square frequency describe the location distribution of the main frequency band, while spectral entropy describes the uniformity and randomness of the frequency component distribution.

[0073] For the measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power After processing the narrowband spectrum, the frequency domain signals are obtained respectively. Frequency domain signal of wheel-rail rolling noise The three indices of centroid frequency, root mean square frequency, and spectral entropy are extracted to obtain the set of input parameters for the neural network.

[0074] The system rapidly infers the rail corrugation depth at the current location using a pre-trained neural network model, specifically including: Based on a pre-trained neural network model, the input parameter set and the rail corrugation wavelength Output the corrugation depth of the rail at the corresponding position. The prediction results constitute the complete rail corrugation shape characteristics at the current moment and current mileage position.

[0075] The pre-trained neural network model is based on a backpropagation neural network and consists of an input layer, several hidden layers, and an output layer. The nodes between the layers are connected by weights. The connection weights are iteratively adjusted using a supervised learning algorithm based on the principle of error backpropagation.

[0076] The training data is based on previously measured axle box vibration acceleration and wheel-rail rolling noise, as well as measured rail corrugation wavelength and depth at corresponding locations. During model training, the learning process of the backpropagation neural network includes two stages: forward propagation and weight update. In the forward propagation stage, the input data is processed sequentially through each layer of the network and converted into output through a nonlinear function. After the output is generated, it is compared with the target value to calculate the error. The error is then propagated back to the network in a reverse manner. The chain rule is used to effectively calculate the gradient of the loss function with respect to each weight, and the adjustment responsibility of each weight is quantified.

[0077] During the weight update phase, gradient descent is used to adjust the weights; the process is iterated to obtain a pre-trained neural network model, which is then used for real-time inference during the online detection process.

[0078] In the S3 wave depth feature extraction of the above-mentioned intelligent online detection method for rail corrugation based on vibration-noise-visual multimodal feature fusion, the final output target is the wave depth of rail corrugation. The main technical approach: S31. Use narrowband spectrum to process the axle box vibration acceleration signal and wheel-rail rolling noise signal within the adaptive window length range at the current moment.

[0079] S32. Use the frequency domain characteristic parameters of vibration and noise within the window length range as real-time input.

[0080] S33. Based on a pre-trained neural network model, quickly infer the rail corrugation depth value at the current position.

[0081] In step S31 above, the parameters for the adaptive window length at the current moment are set as follows: window length 1 second, sliding step size 0.1 seconds, and output frequency 10Hz, i.e., 10 feature updates per second. Under the sampling frequency set in step S1, the number of new sampling points corresponding to a single feature update should not be less than 500 vibration signal points, 2000 noise signal points, and 3 image frames.

[0082] In step S31 above, the measured time series signal of the axle box vibration acceleration is processed. Time series signal of wheel-rail rolling noise sound power S Frequency domain processing primarily utilizes narrowband spectrum as the signal processing method because the characteristic frequencies of the vibrations excited by rail corrugation are narrow relative to the signal bandwidth, and can all be considered as a single narrowband signal. The frequency domain characteristics of acceleration and noise power levels obtained through narrowband spectrum processing can separate and extract excitation signals with similar frequencies and smaller amplitudes from other signals. The main steps and mathematical principles are as follows: Single narrowband signal... Mathematical expression: in, It is the time-varying envelope of the signal, mainly corresponding to amplitude modulation information; It is the time-varying phase of the signal, mainly corresponding to phase modulation information; It is the carrier frequency. Since the cosine function couples information to two sidebands, a real signal is defined for efficient mathematical operations. Analyzed signal It is a complex signal: in, Real signal Performing a Hilbert transform, which involves shifting all frequency components by -90°, achieves the effect of preserving and doubling the positive frequency components of the original signal while completely eliminating negative frequency components. (Analyzing the signal) The spectrum is concentrated at the carrier frequency Nearby, by frequency shifting to baseband, the complex envelope of the signal is obtained. : In the frequency domain, this is represented as a low-pass spectrum concentrated near zero frequency: In the formula The independent variable is the spectrum; For signal complex envelope The spectrum signal after Fourier transform is usually a low-pass spectrum concentrated near zero frequency; To analyze the signal Spectrum Shift to the left The resulting spectral signal.

[0083] The combined envelope is obtained Rectangular coordinate representation: In the formula For signal complex envelope In-phase components, For signal complex envelope The orthogonal components are written as follows: In the formula For signal complex envelope The envelope, For signal complex envelope The phases are written as follows: After the above processing, the measured time series signal of the axle box vibration acceleration was obtained. Time series signal of wheel-rail rolling noise power Each is fully encoded into a low-pass signal. and It contains all the modulation information (amplitude and phase) of the original high-frequency band signal.

[0084] The characteristics of axle box vibration acceleration in both the frequency and time domains are significantly influenced by the second peak frequency of the dynamic wheel-rail force, while wheel-rail noise is largely directly related to the admittance characteristics of the wheel-rail structure and is more sensitive to high-frequency excitation. Both signal types are subject to interference during independent analysis; therefore, a comprehensive consideration of both signal types is necessary to identify rail corrugation. In step S32 above, the frequency domain characteristic parameters after narrowband spectrum processing mainly include three indicators: centroid frequency, root mean square frequency, and spectral entropy. The centroid frequency and root mean square frequency describe the positional distribution of the main frequency band, while the spectral entropy describes the uniformity and randomness of the frequency component distribution. Centroid frequency... The centroid is the "center of gravity" of the signal's frequency spectrum, reflecting the average frequency location where the signal's energy is concentrated. A high centroid frequency means that the main energy of the signal is distributed in the high-frequency region, while a low centroid frequency indicates that the energy is mainly concentrated in the low-frequency region. Its mathematical expression is: in, It is the first One frequency point, It is the first Power at each frequency point. Root mean square frequency. It is an intuitive indicator for measuring the center frequency of a signal's spectral energy distribution. Specifically, it is the weighted average frequency that comprehensively considers both the frequency and the energy at that frequency. Its mathematical expression is: Spectral entropy Spectral entropy is used to measure the uniformity and randomness of the frequency component distribution of a signal. High spectral entropy indicates a relatively uniform frequency distribution, similar to white noise; low spectral entropy indicates the presence of a strong dominant frequency component in the signal. Its mathematical expression is: in: In steps S31 and S32 above, the measured time series signal of the axle box vibration acceleration is obtained. Time series signal of wheel-rail rolling noise sound power After processing the narrowband spectrum, the frequency domain signals are obtained respectively. Time series signal of wheel-rail rolling noise sound power S Then, by extracting the centroid frequency, root mean square frequency, and spectral entropy, the following set of input parameters for the neural network is obtained. It can be used for pre-training and real-time inference.

[0085] in, It is the centroid frequency of the axle box vibration acceleration signal within the window length range. It is the root mean square frequency of the axle box vibration acceleration signal within the window length range. It is the spectral entropy of the axle box vibration acceleration signal within the window length range. It is the centroid frequency of the wheel-rail rolling noise signal within the window length range. It is the root mean square frequency of the wheel-rail rolling noise signal within the window length range. It is the spectral entropy of the wheel-rail rolling noise signal within the window length range.

[0086] In step S33 above, predicting rail corrugation depth using a pre-trained neural network model specifically includes: based on the pre-trained neural network model, the input parameter set... and the rail corrugation wavelength output in step S2 Output the corrugation depth of the rail at the corresponding position. The prediction results constitute the complete rail corrugation shape characteristics at the current moment and current mileage position.

[0087] In step S33 above, the pre-trained neural network model is based on a backpropagation neural network, consisting of an input layer, several hidden layers, and an output layer. Nodes in each layer are connected by weights. Using a supervised learning algorithm, based on the principle of error backpropagation, the connection weights are iteratively adjusted to make the network output as close as possible to the desired output. The training data is based on previously measured axle box vibration acceleration and wheel-rail rolling noise, as well as measured rail corrugation wavelength and depth at corresponding locations. During model training, the backpropagation neural network's learning process includes two stages: forward propagation and weight update. In the forward propagation stage, the input data is processed sequentially through each layer in the network and transformed into output through a nonlinear function. After the output is generated, it is compared with the target value, the error is calculated, and then the error is propagated back to the network in a reverse manner. The chain rule is used to effectively calculate the gradient of the loss function with respect to each weight, thereby quantifying the adjustment responsibility of each weight. In the weight update stage, gradient descent is used to adjust the weights to reduce the error. The above process (from forward propagation to weight update) is repeated on the entire training set until the network performance reaches a satisfactory level or the preset number of iterations is completed. During training, the network's performance is typically evaluated on a separate validation set to avoid overfitting and to tune hyperparameters. Ultimately, a pre-trained neural network model is obtained and used for real-time inference during online detection.

[0088] S4: Based on the wavelength and depth of rail corrugation, determine the key morphological parameters of rail corrugation, compare them with the track maintenance rules, and conduct a quantitative evaluation of rail corrugation defects.

[0089] The quantitative evaluation of rail corrugation defects includes: integrating data correlation, using the mileage position marked by the photoelectric odometer as a reference, binding the output rail corrugation wavelength value and rail corrugation depth value with the mileage position, constructing a three-dimensional data matrix of mileage-depth-wavelength, and aligning the data.

[0090] Based on the rail corrugation wavelength value at the current time or location, and according to the wavelength classification standard, determine whether the defect type at the current time or location is long-wavelength corrugation or short-wavelength corrugation.

[0091] Based on the current rail corrugation depth value at the current time or location, and combined with the determined wavelength properties, if the value exceeds the limit, the rail damage is considered to have reached the level of serious damage.

[0092] Based on the integrated analysis of vibration, noise, and visual data, the system outputs the detection results and judgment conclusions of rail corrugation at the current time or location.

[0093] In the above-mentioned intelligent online detection method for rail corrugation based on the fusion of vibration-noise-visual multimodal features, S4 constructs a fusion decision system based on the wave depth and wavelength characteristic parameters of rail corrugation to achieve quantitative evaluation of rail corrugation defects. This mainly includes the following steps: S41. Integrate and associate data. Using the mileage position marked by the photoelectric odometer as a reference, bind the wavelength value (spatial sequence) output in step S2 and the wave depth value (time sequence) output in step S3 with the mileage position to construct a three-dimensional data matrix of "mileage-wave depth-wavelength". The data alignment error shall not exceed 1 meter.

[0094] S42. Based on the rail corrugation wavelength value at the current time or location output in step S2, determine the type of defect at the current time or location as long-wavelength corrugation (above 100mm) or short-wavelength corrugation (25mm-80mm) according to the wavelength classification standard.

[0095] S43. Based on the rail corrugation depth value at the current time or position output in step S3, and combined with the wavelength properties determined in steps S2 and S42, the amplitude limit for short-wavelength corrugation is 0.1 mm, and the amplitude limit for long-wavelength corrugation is 0.2 mm. If the limit is exceeded, the rail damage is considered to have reached the level of severe damage.

[0096] S44. Based on the integrated vibration-noise-visual data analysis results, output the detection results and judgment conclusions of rail corrugation at the current time or location.

[0097] Multimodal fusion early warning: The wavelength and depth characteristics of rail corrugation obtained from steps S2 and S3 are combined and substituted into... Figure 5 The schematic diagram of the ideal shape of rail corrugation shown can determine the key morphological parameters and basic shape of rail corrugation. A comprehensive comparison with track maintenance rules outputs a status rating and maintenance recommendations. A multi-dimensional indicator fusion decision-making system with graded early warning provides a conclusion on the track condition at the current operating location. After the entire operation is completed, the output results on the distribution of track corrugation and the degree of defect development should be reviewed by experts.

[0098] This embodiment also provides a rail corrugation detection system based on multimodal feature fusion, referring to... Figures 6-8 As shown, it includes: The first part, the hardware architecture, integrates a multimodal edge perception module to construct an on-board detection network; its main components include: Vibration and noise integrated sensing data acquisition module: Deployed in key locations of the train's running gear, the hardware includes a multi-channel data acquisition system, a large-range piezoelectric accelerometer, an acoustic microphone, and data connection cables. Through accelerometers and noise sensors, it collects real-time vibration acceleration data of the bogie axle boxes and noise data of the running gear during normal train operation.

[0099] Visual perception and video image acquisition module: Deployed on the chassis structure near the wheel-rail contact area of ​​the train, the module includes industrial cameras and auxiliary light sources. This module is responsible for acquiring visual signals, capturing high-definition video or images at a high frame rate during train operation to ensure continuous sampling of the rail surface, with a minimum rail surface irregularity wavelength resolution of 10mm.

[0100] The two types of data acquisition modules mentioned above are based on vehicle-mounted multimodal edge perception fusion detection units. They utilize predefined key measurement points on the train running gear and pre-set data acquisition devices to achieve synchronous acquisition and preliminary processing of axle box vibration, wheel-rail noise, rail surface images, and mileage information. They communicate with diagnostic equipment via industrial Ethernet to provide a highly reliable data source for subsequent analysis.

[0101] The second part of the online detection system is the analysis architecture and its diagnostic equipment. The preferred edge computing equipment is responsible for the deep fusion and intelligent analysis of the multimodal data collected by the edge hardware. It integrates computer vision algorithms, advanced signal processing and machine learning algorithms and other multimodal spatiotemporal feature fusion. It performs fusion analysis on visual (rail corrugation wavelength feature extraction in step S2) and vibration noise (rail corrugation wave depth feature extraction in step S3) to realize automatic identification and dynamic evaluation of rail corrugation.

[0102] Data Acquisition Module: This unit is deployed in the train running gear and consists of two main components: a vibration and noise integrated sensing module and a visual sensing module. The vibration and noise integrated sensing module comprises a vibration acceleration sensor and an acoustic microphone, while the visual sensing module consists of a high-frame-rate industrial camera, an LED auxiliary light source, and a photoelectric odometer. The industrial camera has a resolution of 1920×1080ppi and a frame rate of ≥200fps, with the shooting direction perpendicular to the rail surface. The LED auxiliary light source ensures image quality under complex lighting conditions, and the photoelectric odometer outputs real-time mileage information as the wheels rotate, achieving precise matching between the image and spatial position, enabling the system to achieve a minimum rail surface irregularity wavelength resolution of 10mm. The data acquisition module uses an onboard multimodal edge perception fusion detection unit as its core to achieve simultaneous acquisition and preliminary processing of axle box vibration, wheel-rail noise, rail surface images, and mileage information, providing a highly reliable data source for subsequent analysis.

[0103] Rail corrugation wavelength calculation module: This module is Figure 6 The intelligent analysis and diagnostic cloud platform shown is one of the core backend algorithm components. It relies on visual image analysis to achieve precise wavelength measurement. Through four core steps—image preprocessing, target detection, distance calculation, and coordinate transformation—it converts the pixel information of the track surface image into actual wavelength data. This module first processes the track surface video or image (such as...) captured by an industrial camera... Figure 7 (As shown) Preprocessing is performed, and then feature recognition is performed based on a pre-trained model that has learned a large number of track surface images from different scenarios to quickly detect track surface corrugation (such as...). Figure 8 As shown in the figure, the conversion from pixel coordinates to actual spatial coordinates is finally realized. By calculating the distance between adjacent peaks or troughs in the world coordinate system, the actual wavelength of the rail corrugation can be obtained. The wavelength measurement accuracy matches the system's minimum resolution of 10mm.

[0104] Rail corrugation depth calculation module: This module is Figure 6 One of the core backend algorithm components of the intelligent analysis and diagnosis cloud platform shown here achieves accurate wave depth identification based on vibration-noise signal fusion, outputting wave depth values ​​through a closed-loop process of "preprocessing-feature decomposition-model prediction". This module first performs targeted preprocessing on the raw signals transmitted from the edge units, then proceeds to the feature decomposition stage. Using narrowband spectrum, it decomposes the complex vibration and noise signals into effective wave-slip vibrations and other interference components, extracting key spectral parameters such as characteristic frequencies and spectral centroids. Finally, based on the wave depth prediction model, using the characteristic parameters of the vibration spectrum and noise power level as input, it predicts the actual wave-slip depth.

[0105] Track rail condition evaluation module: This module is Figure 6The intelligent analysis and diagnosis cloud platform, as shown, provides key visualizations of its front-end output and decision-making processes. It connects data acquisition and computational execution, achieving quantitative evaluation of rail corrugation conditions through multimodal data fusion. The module primarily aligns and correlates multi-source data. Using real-time mileage information from the photoelectric odometer as a benchmark, it precisely binds the spatial sequence wavelength data output from the wavelength calculation module with the temporal sequence wave depth data output from the wave depth calculation module, constructing a three-dimensional feature matrix of "mileage-wave depth-wavelength." This ensures that each set of corrugation parameters and shapes can be pinpointed to a specific track location. The intelligent analysis and diagnosis cloud platform simultaneously generates early warning information including the location of the fault mileage, corrugation parameters, and risk level, and pushes this information to the maintenance terminal. The platform interface intuitively displays the online status of the equipment, real-time corrugation data, and the visualization results of the three-dimensional feature matrix, providing maintenance personnel with a comprehensive decision-making basis for location and assessment.

[0106] Example 2, refer to Figures 9-11 As an embodiment of the present invention, a method for optimizing the allocation of multiple trades in collaborative construction is provided. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0107] Figure 9 The difference in vertical acceleration of the axle box under corrugated and non-corrugated conditions was compared. When the rail has a certain degree of corrugation, the effective value of the vertical acceleration of the axle box increases from 8 m / s². 2 Increased to 17 m / s 2 The presence of rail corrugation causes a significant increase in the vertical acceleration of the axle box. Furthermore, the wave depth characteristics of rail corrugation are strongly correlated with the vertical vibration acceleration of the axle box. Under the condition of a speed of 60 km / h and a corrugation wavelength of 40 mm, the time-domain characteristic parameters corresponding to different wave depth amplitudes differ as follows: Figure 10 The evaluation parameters involved include peak-to-peak value, rectified average value, and root mean square value. Figure 10 This demonstrates that the generation of corrugation greatly affects the amplitude of the vertical vibration acceleration of the axle box. Furthermore, under the same speed and wavelength, the larger the corrugation amplitude, the larger the amplitude of the axle box vibration acceleration, and the larger the peak-to-peak value, rectified average value, and root mean square value.

[0108] Figure 11The effects of different corrugation amplitudes on the sound power level of wheel-rail rolling noise were compared. A speed of 60 km / h and a corrugation wavelength of 40 mm were selected. The comparative analysis showed that at significant frequencies of rolling noise sound power, the sound power level increased with increasing corrugation amplitude. During the process of increasing the 40 mm short-wavelength corrugation amplitude from 0.01 mm to 0.3 mm, the sound power level at 417 Hz increased from 103 dB to 132 dB, an increase of 28%. Compared to the sound power level at significant frequencies without corrugation, the sound power level at a 40 mm wavelength corrugation amplitude of 0.3 mm increased by 62%. Therefore, rail corrugation significantly increases the sound power level of rolling noise at specific frequencies, and the increase in sound power level becomes more significant with increasing corrugation amplitude.

[0109] Therefore, both axle box vibration acceleration signals and wheel-rail noise signals can reflect the characteristics of rail corrugation excitation to a certain extent. These two types of signals, which are quickly detectable and easily acquired, show a strong correlation between the vibration energy characteristics in the time-frequency domain and the rail corrugation depth level. Using a neural network model, the relationship between vehicle speed, rail corrugation depth characteristics, axle box vibration acceleration level, and wheel-rail rolling noise power level can be established, providing a foundation for neural network analysis and identification of rail corrugation depth characteristics.

[0110] This embodiment also provides a computer device applicable to the rail corrugation detection method based on multimodal feature fusion, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the rail corrugation detection method based on multimodal feature fusion as proposed in the above embodiment.

[0111] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0112] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the rail corrugation detection method based on multimodal feature fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0113] In summary, this invention involves: acquiring high-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area during train operation, and marking mileage positions using a photoelectric odometer; dynamically detecting the real-time acquired video signals, identifying the pixel distance length of the wave-shaped wear wavelength characteristics of the rail surface using a deep learning model, and obtaining the rail corrugation wavelength value through spatial transformation combined with camera calibration parameters; performing second-level analysis and processing of the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal using narrowband spectrum, using the frequency domain feature parameters of vibration and noise as real-time input, and retrieving the rail corrugation depth value at the current position based on a pre-trained neural network model; determining the key morphological parameters of rail corrugation based on the wavelength and depth values ​​of rail corrugation, and conducting a quantitative evaluation of rail corrugation defects by comprehensively comparing them with track maintenance rules.

[0114] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A rail corrugation detection method based on multimodal feature fusion, characterized in that: include, High-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area are collected during train operation and combined with photoelectric odometers to mark mileage positions; Dynamic detection is performed on the real-time acquired video signal. The pixel distance length of the wave-shaped wear wavelength characteristics of the rail surface is identified through a deep learning model. The spatial transformation is combined with the camera calibration parameters to obtain the rail wave wavelength value. The narrowband spectrum is used to perform second-level analysis and processing on the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal. The frequency domain feature parameters of vibration and noise are used as real-time inputs, and the rail corrugation depth value at the current position is inverted based on the pre-trained neural network model. Based on the wavelength and depth of rail corrugation, the key morphological parameters of rail corrugation are determined, and a comprehensive comparison with track maintenance rules is conducted to quantitatively evaluate rail corrugation defects. The method for retrieving the rail corrugation depth value at the current location using the pre-trained neural network model includes using frequency domain feature parameters of vibration and noise within the window length range as real-time input. The frequency domain characteristic parameters after narrowband spectrum processing include three indicators: centroid frequency, root mean square frequency, and spectral entropy. Among them, centroid frequency and root mean square frequency describe the location distribution of the main frequency band, while spectral entropy describes the uniformity and randomness of the frequency component distribution. For the measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power After processing the narrowband spectrum, the frequency domain signals are obtained respectively. Frequency domain signal of wheel-rail rolling noise The three indices of centroid frequency, root mean square frequency, and spectral entropy are extracted to obtain the set of input parameters for the neural network. The system rapidly infers the rail corrugation depth at the current location using a pre-trained neural network model, specifically including: Based on a pre-trained neural network model, the input parameter set and the rail corrugation wavelength Output the corrugation depth of the rail at the corresponding position. The prediction results constitute the complete rail corrugation shape characteristics at the current moment and current mileage position; The pre-trained neural network model is based on a backpropagation neural network and consists of an input layer, several hidden layers, and an output layer. The nodes between the layers are connected by weights. The connection weights are iteratively adjusted using a supervised learning algorithm based on the principle of error backpropagation. The training data is based on previously measured axle box vibration acceleration and wheel-rail rolling noise, as well as measured rail corrugation wavelength and depth at corresponding locations. During model training, the learning process of the backpropagation neural network includes two stages: forward propagation and weight update. In the forward propagation stage, the input data is processed sequentially through each layer of the network and converted into output through a nonlinear function. After the output is generated, it is compared with the target value to calculate the error. The error is then propagated back to the network in a reverse manner. The chain rule is used to effectively calculate the gradient of the loss function with respect to each weight, and the adjustment responsibility of each weight is quantified. During the weight update phase, gradient descent is used to adjust the weights; the process is iterated to obtain a pre-trained neural network model, which is then used for real-time inference during the online detection process.

2. The rail corrugation detection method based on multimodal feature fusion as described in claim 1, characterized in that: The acquisition of high-definition video signals of axle box vibration acceleration, wheel-rail noise, and rail surface area during train operation includes using a single bogie as a measurement unit, with the measurement points of each unit arranged as follows: The axle box vibration acceleration is measured at the center of the axle box cover surface of the left and right wheels, with one vertical measurement direction set on each side; the wheel-rail noise is measured near the wheels on both sides of the bogie, with one acoustic measurement point set on each side, and the distance from the wheel-rail contact point is less than the maximum measurement distance. High-definition video of the track surface area is captured from the bottom of the bogie, directly facing the track surface area, with one video monitoring point set on each side; the photoelectric odometer is set at the end of the non-drive wheel axle of the bogie, with a total of one measuring point, and the measured value is regarded as the travel distance of the whole vehicle.

3. The rail corrugation detection method based on multimodal feature fusion as described in claim 2, characterized in that: The pixel distance length for identifying the wavelength characteristics of track surface wavy wear through a deep learning model includes, before the detection begins, setting up an image acquisition system and calibrating the camera. High-definition videos of the left and right side rail surfaces are acquired in real time during train operation; Target detection is performed using a pre-trained deep learning model, and the recognition unit is the bounding rectangle of a single valley edge contour region. Based on the pixel coordinates of the centers of adjacent detection frames for rail corrugation detected by the deep learning model, and combined with the camera calibration results before each detection, a spatial coordinate transformation is performed to obtain the true distance, which is denoted as the wavelength value. .

4. The rail corrugation detection method based on multimodal feature fusion as described in claim 3, characterized in that: The process of obtaining the true wavelength value by combining camera calibration parameters with spatial transformation includes establishing a vertically shooting visual inspection system: a high-performance industrial camera is located directly above the rail, with the camera's lens axis perpendicular to the rail surface, focusing and shooting vertically downwards; The camera calibration adopts the Zhang Zhengyou calibration method to obtain the camera's internal and external parameters, including focal length, principal point coordinates, rotation matrix, and translation vector, and to establish the mapping relationship between the pixel coordinate system and the world coordinate system. Based on the dataset constructed from high-resolution images of rail corrugation and labeled rail corrugation detection boxes, a target detection model for rail corrugation recognition is obtained through deep learning training; both the labeled detection boxes and the detection boxes output during actual operation are bounded rectangles of the contour region of each corrugation trough. The pixel coordinate results of the centers of adjacent detection boxes for rail corrugation returned by the deep learning model specifically include: During online detection, a trained model is used for real-time inference, outputting the detected data. A set of valley bounding boxes is generated. For each valley bounding box, the pixel coordinates of the center point are calculated. All center points are sorted according to their horizontal coordinates to obtain an ordered center point sequence. Based on the ordered center point sequence, the pixel distance between adjacent valleys and the average wavelength are calculated. After the average wavelength is calculated, the detection quality of the image frame is evaluated. Combining the camera calibration results before each detection, and based on the deep learning model, the pixel coordinates of the centers of adjacent detection boxes of rail corrugations are detected and mapped to convert the pixel length into the actual rail corrugation wavelength. By detecting each continuous image frame extracted from the online video stream or video, the time series of rail corrugation wavelengths during train travel is obtained, which is the information sequence of rail corrugation wavelength and track mileage.

5. The rail corrugation detection method based on multimodal feature fusion as described in claim 4, characterized in that: The second-level analysis and processing of the real-time acquired axle box vibration acceleration signal and wheel-rail rolling noise signal using narrowband spectrum includes: performing signal processing on the axle box vibration acceleration signal and wheel-rail rolling noise signal within the adaptive window length range at the current moment using narrowband spectrum; and setting the adaptive window length for the current moment. For the measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Perform frequency domain processing; For a single narrowband signal, since the cosine function couples information to two sidebands, the analytic signal of the real signal is defined as a complex signal: The spectrum of a complex signal is concentrated near the carrier frequency. By frequency shifting to the baseband, the complex envelope of the signal can be obtained. ; Obtain complex envelope Represented in rectangular coordinates, it is denoted in the frequency domain as a low-pass spectrum concentrated near zero frequency. ; Measured time series signal of axle box vibration acceleration Time series signal of wheel-rail rolling noise sound power Each signal is fully encoded into a low-pass signal, containing all the modulation information of the original high-frequency narrowband signal.

6. The rail corrugation detection method based on multimodal feature fusion as described in claim 5, characterized in that: The quantitative evaluation of rail corrugation defects includes: integrating data correlation, using the mileage position marked by the photoelectric odometer as a reference, binding the output rail corrugation wavelength value and rail corrugation depth value with the mileage position, constructing a three-dimensional data matrix of mileage-depth-wavelength, and aligning the data. Based on the rail corrugation wavelength value at the current time or location, and according to the wavelength classification standard, determine whether the defect type at the current time or location is long-wavelength corrugation or short-wavelength corrugation. Based on the current rail corrugation depth value at the current time or location, and combined with the determined wavelength properties, if the limit is exceeded, the rail damage is considered to have reached the level of serious damage. Based on the integrated analysis of vibration, noise, and visual data, the system outputs the detection results and judgment conclusions of rail corrugation at the current time or location.

7. A rail corrugation detection system based on multimodal feature fusion, based on the rail corrugation detection method based on multimodal feature fusion as described in any one of claims 1 to 6, characterized in that: The first part comprises a hardware architecture that integrates multimodal edge perception modules to construct an onboard detection network; its main components include: Vibration and noise integrated sensing data acquisition module: Deployed in key locations of the train running gear, it is equipped with a multi-channel data acquisition system, a large-range piezoelectric accelerometer, an acoustic microphone, and a data connection cable; Through the accelerometer and noise sensor, it collects vibration acceleration data of the bogie axle box and noise data of the running gear in real time during normal train operation; Visual perception and video image acquisition module: Deployed on the frame structure near the wheel-rail contact area of ​​the train, it is equipped with an industrial camera and auxiliary light source; it is responsible for acquiring visual signals and capturing high-definition video or images at a high frame rate during train operation to ensure continuous sampling of the rail surface; The acquisition module is based on an on-board multimodal edge perception fusion detection unit. It utilizes predefined key measurement points of the train running gear and pre-set data acquisition devices to achieve synchronous acquisition and preliminary processing of axle box vibration, wheel-rail noise, rail surface images and mileage information. The second part is the analysis architecture and diagnostic equipment of the online detection system, which is responsible for the deep fusion and intelligent analysis of multimodal data collected by edge hardware, integrating multimodal spatiotemporal feature fusion, and performing fusion analysis of visual and vibration noise to achieve automatic identification and dynamic evaluation of rail corrugation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the rail corrugation detection method based on multimodal feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the rail corrugation detection method based on multimodal feature fusion as described in any one of claims 1 to 6.

Images