Method for extracting spatial displacement of railway track fastening

Abstract

The application discloses a railway track fastener space displacement extraction method based on camera calibration, first, a fixed camera is used to shoot plane calibration boards in different positions and postures, camera internal parameters and external parameter matrixes are calculated, then camera radial distortion modeling and calculation are carried out, camera internal and external parameters and distortion are iteratively calculated by using an LM algorithm, finally, a fastener two-dimensional image displacement extraction model is established, fastener two-dimensional image displacement is converted into three-dimensional space displacement, and the algorithm effectiveness is verified in a laboratory. The application can be used as a component of a railway track fastener health monitoring real-time early warning subsystem, and can directly and timely identify whether the space position and displacement of the railway track fastener are abnormal. The application improves the automation, intelligence, accuracy and robustness of intelligent identification of the railway track fastener health monitoring real-time early warning subsystem, and provides a solution for establishment of a railway track fastener position and space displacement online real-time early warning subsystem.

Description

Technical Field

[0001] This invention relates to a method for monitoring the health of railway track fasteners, specifically a method for identifying and extracting the spatial displacement of railway track fasteners based on camera calibration. Background Technology

[0002] To date, my country's total railway operating mileage exceeds 154,900 kilometers. After nearly a decade of large-scale construction, my country's railway tracks will enter a comprehensive maintenance period. Track inspection has always been a key research topic in the field of rail transit, and the condition inspection of track fasteners is an important part of track inspection tasks.

[0003] Railway infrastructure, including tracks, is constantly exposed to the natural environment and subjected to extreme weather conditions year-round. Therefore, changes in track shape and damage to connecting devices consistently impact the safety of railway line operations. Furthermore, the contact friction and vibration between wheels and tracks can easily cause defects such as rail breakage or missing fasteners. Once fasteners are defective or missing, they can cause horizontal or longitudinal movement of the rails, potentially leading to major accidents such as train derailments. Therefore, rapidly detecting the position and spatial displacement of rail fasteners and assessing their health status is crucial. Safety inspection of railway infrastructure is a key aspect of ensuring safe railway operations, and the condition inspection of rail fasteners is a vital link, holding significant practical importance for ensuring the safety and stability of railway transportation.

[0004] Currently, there are two main methods for inspecting track fasteners: (1) manual inspection and (2) track inspection vehicle inspection. However, both methods have inherent drawbacks. For example, manual inspection involves long maintenance mileage, high maintenance costs, and is labor-intensive, making it prone to missed inspections and difficult to meet the needs of the rapidly developing railway system. As for track inspection vehicle inspection, the inspection cycle is long, and the inspection process occupies normally operating track lines, resulting in low inspection efficiency.

[0005] Therefore, how to achieve high-precision and rapid detection of the position and spatial displacement of track fasteners is a problem that urgently needs to be solved. Research on methods for identifying railway track fasteners and extracting spatial displacement has important practical significance. Summary of the Invention

[0006] To address the problems of long maintenance mileage, high maintenance costs, heavy workload, and high risk of missed inspections in traditional manual inspection methods, which are difficult to meet the needs of the rapid development of the railway system, and the low inspection efficiency of track inspection vehicles due to their long inspection cycle and occupation of normal operating track lines, this invention provides a method for extracting the spatial displacement of railway track fasteners.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for extracting the spatial displacement of railway track fasteners includes the following steps:

[0009] Step 1: Take pictures of the planar calibration board at different positions and orientations using a fixed camera. Based on the basic matrix theory, determine the internal and external parameters of the camera by measuring the geometric relationship between the camera and the calibration board.

[0010] Step 2: Considering the significant radial distortion, after establishing the radial distortion expression, the initial distortion value is obtained using the least squares method; the reprojection error is minimized by using the Levenberg-Marquardt (LM) algorithm to calculate the actual pixel coordinates and the calculated pixel coordinates, and the camera's intrinsic and extrinsic parameters and distortion coefficients are iterated repeatedly until convergence, thereby improving the instrument's measurement accuracy.

[0011] Step 3: Establish a mathematical model for the two-dimensional image displacement extraction of railway track fasteners; introduce the concept of a cyclic matrix to perform dense sampling on the area around the track fasteners to generate a fastener sample set; use a Gaussian kernel function to map the fastener features to a high-dimensional space to make them linearly separable; calculate the maximum value of the response function, which is the area where the fastener is located in the next frame, thus completing the extraction of the two-dimensional image displacement of the railway track fasteners; use the correspondence between the two-dimensional image and the three-dimensional real coordinates established in Step 1 and Step 2 to convert the two-dimensional image displacement of the fasteners into the three-dimensional real space displacement of the fasteners;

[0012] Step 4: Simulate various working conditions during construction using laboratory equipment, conduct long-term monitoring of the track using cameras set up in different locations, collect data using high-definition cameras equipped with zoom lenses and high-precision targets of multiple sizes, and extract the spatial displacement of the track fasteners using the methods from Step 1 to Step 3.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] 1. This invention extracts the spatial displacement of track fasteners using a visual perception method. It uses cameras set up in various orientations to monitor the track over a long period of time and explores the applicability of the track fastener spatial displacement extraction algorithm under bright light conditions.

[0015] 2. This invention is based on fastener tracking technology. It uses a cyclic matrix to perform dense sampling of the area around the fastener, and transforms the filter training process from the spatial domain to the frequency domain after Fourier transform, thus converting the matrix inversion operation into the vector dot product operation.

[0016] 3. This invention can serve as a component of a real-time health monitoring and early warning subsystem for railway track fasteners, directly and in real-time identifying and determining whether the spatial position and displacement of railway track fasteners are abnormal. This invention improves the automation, intelligence, accuracy, and robustness of the intelligent identification capabilities of the real-time health monitoring and early warning subsystem for railway track fasteners, providing a solution for establishing an online real-time early warning subsystem for the position and spatial displacement of railway track fasteners. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention;

[0018] Figure 2 This is a schematic diagram of correlation filtering;

[0019] Figure 3 Correlation filtering is applied to training and detection;

[0020] Figure 4 Example of vertical cyclic displacement of the base sample;

[0021] Figure 5 This serves as a test field for track vibration reduction.

[0022] Figure 6 The results of corner point identification for working conditions A1-A6;

[0023] Figure 7 This is the layout diagram for test condition B1;

[0024] Figure 8 The results are from the B1 test.

[0025] Figure 9 This is the layout diagram for the B2 test condition.

[0026] Figure 10 The results are from the B2 test.

[0027] Figure 11 This is the layout diagram for test condition B3;

[0028] Figure 12 The results are from the B3 test.

[0029] Figure 13 This is the layout diagram for test condition B4;

[0030] Figure 14 The results are from the B4 test. Detailed Implementation

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0032] This invention provides a method for extracting the spatial displacement of railway track fasteners based on camera calibration. First, a fixed camera is used to photograph a planar calibration plate at different positions and orientations. The camera's intrinsic and extrinsic parameter matrices are calculated. Then, radial distortion of the camera is modeled and calculated. The LM algorithm is used to iteratively calculate the camera's intrinsic and extrinsic parameters and distortion. Finally, a two-dimensional image displacement extraction model for the fastener is established, converting the two-dimensional image displacement into three-dimensional spatial displacement. The effectiveness of the algorithm is verified in the laboratory. Figure 1 As shown, the method specifically includes the following steps:

[0033] Step 1: Using a fixed camera, photograph the planar calibration board at different positions and orientations. Based on fundamental matrix theory, determine the camera's internal and external parameters by measuring the geometric relationship between the camera and the calibration board. This includes the following steps:

[0034] Step 1: The coordinates of each marked point i on the planar target are represented homogeneously in the world coordinate system, denoted as (U... i V i 1) T The pixel coordinate system is denoted as (u i v i 1) T Let A be the camera intrinsic parameter matrix, and let the extrinsic parameter matrix P have its first two columns denoted as R1 and R2, and its last column as T1. i Given the scaling factor, we can obtain formula (1):

[0035]

[0036] Let A(R1 R2 T1) be denoted as the homography matrix H, and let the nine elements of matrix H be H... 11 -H 33 Let the first three columns of H be denoted as (H1, H2, H3). Substituting them into equation (1) yields:

[0037]

[0038] Step 12: Eliminate z from equation (2) i We can obtain:

[0039]

[0040] In the formula, H is a non-full-rank homogeneous matrix with 8 independent unknown elements. Since each pair of markers can provide two constraint equations, when the number of markers on an image is greater than or equal to 4, the corresponding homogeneous matrix H can be obtained by least-squares linear solution. Since R1 and R2 satisfy the unity orthogonality relation, then:

[0041] R1 T R2=0 (4)

[0042] R1 T R1 = R2 T R2=1 (5)

[0043] Based on the relationship between H and R, we can know that:

[0044] R1 = A -1 H1,R2=A -1 H2 (6)

[0045] Step 13: Substituting equation (6) into (4) and (5) yields:

[0046]

[0047] H1 T A -T A -1 H1 = H2 T A -T A -1 H2=1 (8)

[0048] Let A -T A -1 =B, and the nine elements of B are denoted as B0. 11 -B 33 Let α, β, and γ be three equivalent parameters, and u0 and v0 be the coordinates of the center point of the imaging plane in the pixel coordinate system, respectively. We can obtain:

[0049]

[0050]

[0051] From equation (9), we can see that B is a symmetric matrix, and its six-dimensional vector representation is:

[0052] b = [B 11 B 12 B 22 B 13 B 23 B 33 ] T (11)

[0053]

[0054] In the formula

[0055]

[0056] Step 14: The basic constraint equations for the camera's intrinsic parameters can be rewritten as follows:

[0057]

[0058] Since the homogeneous matrix H is known, and matrix v is composed of elements of H, matrix v can be obtained. Each calibration image provides two constraint equations, and b has six unknowns; therefore, b can be obtained from three calibration images. When there are more than three calibration images, the least squares method is used to solve for the optimal value. The camera's intrinsic parameters can be expressed as:

[0059]

[0060] After obtaining the camera's intrinsic parameters, the extrinsic parameters are then calculated. Based on A(R1 R2 T) = H, the (R1 R2 T) corresponding to each sample image is obtained. According to the spatial orientation relationship of R1, R2, and R3, we can obtain:

[0061] R3 = R1 × R2 (16)

[0062] This allows us to obtain the lens's intrinsic and extrinsic parameters.

[0063] Step 2: Considering the significant radial distortion, after establishing the radial distortion expression, the initial distortion value is obtained using the least squares method. The Levenberg-Marquardt (LM) algorithm is then used to minimize the reprojection error between the actual pixel coordinates and the calculated pixel coordinates. This process is iterated repeatedly with the camera's intrinsic and extrinsic parameters and distortion coefficients until convergence, thereby improving the instrument's measurement accuracy. Specifically, the steps include:

[0064] Step 21: Calibrate lens distortion. Based on Zhang Zhengyou's calibration method, consider the radial distortion, which has a significant impact on the distortion model:

[0065]

[0066] In the formula (x,y) and These represent the normalized image coordinates for the undistorted and distorted images, respectively, where r is the distance from the pixel to the image center point, i.e., r 2 =x 2 +y 2 k1 and k2 are distortion coefficients. The transformation relationship between coordinates (u,v) in the image coordinate system and coordinates in the pixel coordinate system is as follows:

[0067]

[0068] Step 22: In the pixel coordinate system of the distorted image expression:

[0069]

[0070] Substituting into the radial distortion formula, we get:

[0071]

[0072] Simplified to:

[0073]

[0074] Represented in matrix form:

[0075]

[0076] Let D be the left half of the above equation and k be the right half, for a simplified representation:

[0077] Dk=d (23)

[0078] The radial distortion coefficient is solved using least squares:

[0079]

[0080] Steps 2 and 3: At this point, the camera's intrinsic parameters, extrinsic parameters, and distortion coefficients have all been determined. The direct results can only be used as initial values. Next, the reprojection error is calculated based on the actual pixel coordinates and the calculated pixel coordinates. The reprojection error is optimized by minimizing the value, iterating through the camera's intrinsic and extrinsic parameters and distortion coefficients until convergence.

[0081] Zhang Zhengyou's calibration method establishes the relationship between three-dimensional world coordinates and two-dimensional pixel coordinates by using a fixed camera to photograph a planar calibration plate at different positions and orientations. The intrinsic, extrinsic, and distortion coefficients of the camera, obtained through linear solutions, are optimized using the Levenberg-Marquardt algorithm for nonlinear parameters, resulting in high theoretical accuracy and improving the measurement accuracy of instruments such as optical navigation systems.

[0082] Step 3: Establish a mathematical model for the two-dimensional image displacement extraction of railway track fasteners; introduce the concept of a cyclic matrix to perform dense sampling of the area surrounding the track fasteners, generating a fastener sample set; use a Gaussian kernel function to map the fastener features to a high-dimensional space to make them linearly separable; calculate the maximum value of the response function, which is the region where the fastener is located in the next frame, thus completing the extraction of the two-dimensional image displacement of the railway track fasteners; utilize the correspondence between the two-dimensional image and the three-dimensional real coordinates established in Steps 1 and 2 to convert the two-dimensional image displacement of the fasteners into the three-dimensional real space displacement of the fasteners. Specifically, this includes the following steps:

[0083] Step 3: 1. Establish a mathematical model for displacement extraction from two-dimensional images of railway track fasteners. This involves designing a filter; the location of the maximum response when the filter is applied to the fastener area is the location of the fastener. For example... Figure 2 As shown.

[0084] The filter is represented by the following formula:

[0085]

[0086] In the formula, g represents the correlation output, f represents the input image, and h represents the filter template. According to the convolution theorem in Fourier transform, the following form is obtained in the frequency domain:

[0087] G = F·H * (28)

[0088] In the formula, G, F, and H correspond to the Fourier transforms of g, f, and h, respectively.

[0089] As a discriminative method, this fastener tracking algorithm is trained using frame t as the training set to obtain the correlation filter corresponding to the current frame. Then, correlation operations are performed with the image of frame t+1. After obtaining the maximum response, an inverse Fourier transform is performed to obtain the predicted position of the fastener in frame t+1. Figure 3 As shown.

[0090] During the training phase, the fastener tracking algorithm obtains the relevant filter through ridge regression training. Let x be the input representing the fastener image. i The training filter weights are ω i The output can be expressed as:

[0091] f(x i )=ω T x i (29)

[0092] At this point, the tracking problem is transformed into a ridge regression problem in machine learning, with the objective function being:

[0093]

[0094] In the formula, y i To set the labels of the training samples, λ is the penalty term coefficient to avoid overfitting. Equation (30) has a closed-form solution, which can be written in matrix form and generalized to the complex domain to obtain:

[0095] ω=(X H X+λI) -1 X H y (31)

[0096] In the formula, X H Let X be the conjugate transpose of X.

[0097] Step 3.2: For an n×n matrix, the time complexity of inverting it is O(n^2). 3 The computational complexity is enormous, making it unsuitable for online learning. To reduce computational complexity, the concept of a cyclic matrix is ​​introduced. The training samples for this fastener tracking algorithm are obtained by cyclically shifting the fastener samples. Taking a one-dimensional feature as an example, assuming the fastener sample is an n×1 dimensional vector, represented as x=(x1,x2,…x…) n ) TDefine the vertical shift matrix P:

[0098]

[0099] Obviously there are:

[0100] Px=(x n ,x1,...x n-1 ) T (33)

[0101] It's not hard to see that Px shifts x by one element in the vertical direction, while P... μ x is then shifted by μ elements. Clearly, the positive and negative sample sets can be obtained by repeatedly multiplying the base sample x by matrix P. For the two-dimensional case, the horizontal displacement matrix can be defined similarly, which will not be elaborated further. The effect of the vertical cyclic displacement of the base samples is as follows... Figure 4 As shown.

[0102] Arrange the column vectors in the shifted sample set by row to obtain the circular matrix X, as shown in equation (34):

[0103]

[0104] Based on the properties of circulant matrices, diagonalization is performed using the Discrete Fourier Transform, expressed as:

[0105]

[0106] In the formula, Let x be the discrete Fourier transform of vector x. F is the Fourier transform matrix of the corresponding order. Substituting equation (35) into equation (31), we note that F is a unitary matrix. Since inverting a cyclic matrix is ​​equivalent to inverting its eigenvalues, we can obtain the following using the above properties:

[0107]

[0108] In the formula, This represents the convolution operation. Using the properties of circular matrix convolution, we can obtain the frequency domain expression of equation (36):

[0109]

[0110] In the formula, ⊙ represents the dot product operation. In summary, this fastener tracking algorithm, by introducing a cyclic matrix and utilizing its favorable properties, transforms the inversion operation into a vector dot product operation, significantly improving the computation speed.

[0111] Step 3: The above derivation is based on linearly separable cases. However, in real life, many situations involve linearly inseparable cases. Therefore, a kernel function is introduced, using nonlinear mapping... Mapping to a high-dimensional feature space achieves linear separability. Let:

[0112]

[0113] Equation (29) can be rewritten as:

[0114]

[0115] The problem is transformed into finding the α values ​​for each term. i The optimal solution to the problem can be obtained by taking the derivative, and can be written in matrix form:

[0116] α * =(K+λI) -1 y (40)

[0117] In the formula, α * Let α be the optimal solution matrix, and K be the kernel matrix composed of sample dot products {K}. ij |K ij =k(x i ,x j )}, where y represents each term y i The matrix formed.

[0118] It can be proven that K is also a circular matrix, and based on the above, the optimal solution in the frequency domain can also be found:

[0119]

[0120] Steps 3 and 4: In the detection phase, after obtaining the filter for the current frame through training, the fastener tracking algorithm will perform dense sampling in the next frame image with the location of the fastener in the current frame as the center, using a sliding window traversal method to form a sample set z.

[0121] The sample z is processed using the filter of the current frame, and the result is as follows:

[0122]

[0123] In the formula, α is the filter obtained by training in the current frame, and k(z,x) i Let ) represent the kernel function. After Fourier transform and transfer to the frequency domain, we have:

[0124]

[0125] In the formula, K xz K is the kernel matrix. z =k(z) i ,x i The algorithm uses the baseline row vector of the response function, and since only basic samples are needed in the calculation, the algorithm is very fast. The detection area corresponding to the maximum value of the response function is the fastener area in the next frame.

[0126] After each frame of image detection is completed, in order to adapt to changes in the appearance of the fastener object during subsequent tracking, the fastener tracking algorithm updates the filter template. The update formula is determined by equations (44) and (45):

[0127] x t =(1-η)x t-1 +ηx′ (44)

[0128] α t =(1-η)α t-1 +ηα′ x (45)

[0129] In the above update formula, x′ represents the currently detected sample, x t and x t-1 These represent the fastener samples in the current frame and the previous frame, respectively, α′ x α represents the coefficients obtained during training when the input sample is x′. t and α t-1 These represent the coefficients when the response function is at its maximum in the current frame and the previous frame, respectively, and η is the learning rate.

[0130] Step 35: Using the correspondence between the two-dimensional image and the three-dimensional real coordinates established in Step 1 and Step 2, convert the displacement of the two-dimensional image of the fastener into the three-dimensional real spatial displacement of the fastener.

[0131] Step 4: Simulate various working conditions during construction using laboratory equipment. Conduct long-term track monitoring using cameras mounted at different locations. Employ high-definition cameras equipped with zoom lenses and high-precision targets of multiple sizes to collect data, verifying the effectiveness of the track fastener spatial displacement extraction algorithm based on fastener tracking technology. Specifically, this includes the following steps:

[0132] Step 41: Simulate various working conditions during construction using laboratory equipment. Use cameras set up in different locations for long-term monitoring of the track. Use high-definition cameras equipped with zoom lenses and high-precision targets of multiple sizes to collect data. Verify the effectiveness of the track fastener spatial displacement extraction algorithm based on fastener tracking technology. Identify the track displacement response under static and dynamic loads. Simulate the distance of the camera installation position by changing the lens focal length. Simulate the situation of identifying track displacement when there is interference in the foreground.

[0133] Step 4.2: Based on the method proposed in Steps 1 to 3 of this invention, the accuracy of the algorithm of this invention is verified by comparing the reconstructed and measured orbital displacements.

[0134] Example:

[0135] This embodiment simulates various working conditions during construction using laboratory equipment. Long-term track monitoring is conducted using cameras positioned at different locations. High-definition cameras equipped with zoom lenses and high-precision targets of multiple sizes are employed to collect data. Combining vision-based fastener tracking and line recognition technologies, the influence of hardware selection, deployment location, and deformation recognition accuracy under different static and dynamic load conditions is explored. Based on the algorithm proposed in this invention, the accuracy of the algorithm is verified by comparing the reconstructed and measured track displacements.

[0136] This experiment was conducted at the Research and Development Institute of the Civil Engineering Department of China Railway Design Group Co., Ltd., and the Vibration and Noise Reduction Laboratory of the National Engineering Research Center for Digital Construction and Evaluation Technology of Urban Rail Transit. The laboratory has a track vibration reduction test field with a total track structure length of 29 meters. Initially, it includes a floating slab track bed, an integral track bed, and trapezoidal sleepers. It is equipped with an EES-70W pure wheel-rail contact electromagnetic excitation system, capable of providing a rated sinusoidal excitation force of 70kN, a rated impact excitation force of 210kN, a continuous excitation force frequency of 1–1000Hz, synchronous vertical downward excitation of the rails by four wheels, and a guide transmission speed of 1cm / s in the non-excitation state, precisely locating the excitation point. Continuous forces include random, sinusoidal, and sinusoidal plus random; impact types include half-sine, sawtooth wave, triangular wave, rectangular, and trapezoidal wave. The experiment also used a high-definition camera equipped with a zoom lens and high-precision targets of multiple sizes to collect data, while contact displacement gauges and strain gauges were used to synchronously collect data on the structural response. A schematic diagram of the track vibration reduction test field is shown below. Figure 5 As shown in the image.

[0137] This experiment targets the key technical problem of track deformation recognition based on computer vision technology. Taking into account the specific conditions of the laboratory, two experimental schemes were set up: a three-dimensional deformation recognition experiment and a track displacement recognition experiment. For the two experiments, six experimental conditions with fixed-focus bright light and four experimental conditions with short-focus quasi-static load, short-focus dynamic load, long-focus dynamic load, and foreground interference were set up to verify the effectiveness and accuracy of the algorithm of the present invention.

[0138] Step 1: Long-term monitoring of the track was conducted using six cameras positioned at different locations (conditions A1-A6) to explore the applicability of the track 3D deformation recognition algorithm under bright light conditions. A 10×7 square checkerboard (square side length 42mm) was used to calibrate the monitoring cameras set in fixed positions. Preliminary calculations were performed by identifying the checkerboard corner points in multiple images. The corner point finding results are as follows: Figure 6 As shown in Table 1, the intrinsic parameter matrix of the camera under various operating conditions is calculated using a calibration algorithm, and the extrinsic parameter matrix is ​​shown in Table 2.

[0139] Table 1. Camera intrinsic parameter matrix under operating conditions A1-A6 (unit: m)

[0140]

[0141] Table 2. Extrinsic parameter matrices of the camera under operating conditions A1-A6 (unit: m)

[0142]

[0143] Step 2: The distortion coefficients of the camera are calculated using a calibration algorithm. The distortion coefficients of the camera under various working conditions are shown in Table 3. The maximum reprojection error calculated is 4.2%, and the calibration result meets the accuracy requirements. Based on the calibrated camera intrinsic and extrinsic parameter matrix and distortion coefficients, the three-dimensional coordinates of the verification feature points are calculated. The calculated grid side lengths and errors are shown in Table 4. Table 4 shows a maximum error of 2.20%, proving the accuracy of the algorithm of this invention.

[0144] Table 3. Camera distortion coefficients under operating conditions A1-A6

[0145] Operating Condition A1 [[2.47405691e+00],[-4.30626547e+00],[1.49802283e-01],[-3.75261198e-02],[-1.90180294e+03]] Operating Condition A2 [[-1.64414349],[15.13780174],[-0.06224365],[0.07152748],[9.15188976]] Operating Condition A3 [[-1.19017184e+00],[1.93563079e+02],[-1.34458799e-02],[-1.73458315e-02],[8.21513164e+01]] Operating Condition A4 [[-0.62847077],[1.84755018],[0.00758155],[0.03058512],[-2.16991317]] Operating Condition A5 [[-4.73889434e-01],[7.96197528e-01],[-1.86975641e-05],[-1.83599651e-02],[-7.21082027e-01]] Operating Condition A6 [[-4.80055453e+00],[2.43620772e+03],[1.17720827e-01],[3.39573228e-02],[4.37262797e+00]]

[0146] Table 4 shows the predicted grid side lengths and relative errors under operating conditions A1-A6.

[0147] project Grid side length (mm) relative error Operating Condition A1 41.987 0.03％ Operating Condition A2 42.681 1.62％ Operating Condition A3 42.063 0.15％ Operating Condition A4 42.927 2.20％ Operating Condition A5 42.334 0.79％ Operating Condition A6 41.568 1.03％

[0148] Step 3: Track displacement recognition algorithm based on fastener tracking technology. Track displacement is identified through visual perception. Conditions B1 and B2 identify track displacement responses under static and dynamic loads, respectively. Condition B3 simulates the distance of the camera installation position by changing the lens focal length. Condition B4 simulates the situation where track displacement is identified when there is interference in the foreground. A total of 4 experimental conditions were set.

[0149] Step 4: For load case B1, the self-weight of the electromagnetic excitation system is used as a load to simulate a quasi-static load condition by moving it at low speed on the track. The experimental setup for load case B1 is as follows: Figure 7 As shown, the experimental results are as follows: Figure 8 As shown, the light-colored line represents the result obtained using a vision-based method, while the dark-colored line represents the result obtained using an LVDT displacement sensor. It can be seen that the vision-based method successfully acquired and identified the track displacement under this condition, with an accuracy of 0.01mm. The identified track displacement shows the same trend as the displacement acquired by the LVDT, but the value is slightly larger. The reason for this is that the LVDT base is placed on a floating platform, and it acquires the relative displacement between the track and the floating platform, while the vision-based acquisition device acquires the displacement of the track relative to the ground. Therefore, the slightly larger value is reasonable.

[0150] For load case B2, an electromagnetic excitation system was used to simulate a dynamic load condition by in-situ excitation (10Hz). For load case B3, the excitation force was increased compared to load case B2. The experimental setup for load cases B2 and B3 is shown in Figure 9. Figure 11 As shown, the experimental results are as follows: Figure 10 , Figure 12 As shown, the light-colored line represents the result obtained using a vision-based method, while the dark-colored line represents the result obtained using an LVDT displacement sensor. It can be seen that the vision-based method successfully acquired and identified the track displacement under this condition, with an accuracy of 0.01mm. The identified track displacement shows the same trend as the displacement acquired by the LVDT, but the value is slightly larger. The reason for this is that the LVDT base is placed on a floating platform, and it acquires the relative displacement between the track and the floating platform, while the vision-based acquisition device acquires the displacement of the track relative to the ground. Therefore, the slightly larger value is reasonable.

[0151] For load case B4, an electromagnetic vibration system was used to simulate dynamic load conditions through in-situ excitation (10Hz), with railing obstruction simulating foreground interference. The experimental setup for load case B4 is as follows. Figure 13 As shown, the experimental results are as follows: Figure 14 As shown in the diagram. The light-colored line represents the result obtained using a vision-based method, while the dark-colored line represents the result obtained using an LVDT displacement sensor. It can be seen that the vision-based method successfully acquired and identified the track displacement under this condition, with an accuracy of 0.01mm. The identified track displacement shows the same trend as the displacement acquired by the LVDT, although the value is slightly larger. The reason for this is that the LVDT base is placed on a floating platform, and it acquires the relative displacement between the track and the floating platform, while the vision-based acquisition device acquires the displacement of the track relative to the ground. Therefore, the slightly larger value is reasonable.

[0152] This experiment, based on the results of camera calibration and 3D coordinate calculation, combined with fastener tracking and line recognition technologies, explores the influence of hardware selection, deployment location, and different static and dynamic load conditions on deformation recognition accuracy. In summary, the accuracy of track displacement recognition can reach 0.01mm, indicating a good match between the reconstructed value and the true value, thus proving the effectiveness and accuracy of the proposed method.

Claims

1. A method of extracting spatial displacement of a railway track fastening, characterized in that The method comprises the following steps: Step one, taking different positions and attitudes of the planar calibration board by fixed camera, based on the basic matrix theory, the internal and external parameters of the camera are determined by measuring the geometric relationship between the camera and the calibration board; Step two, considering the radial distortion which has greater influence, the radial distortion expression is established, and the least square method is used to obtain the initial value of the distortion; the internal and external parameters of the camera and the distortion coefficient are repeatedly iterated until convergence through Levenberg-Marquardt algorithm to minimize the re-projection error calculated by the actual pixel coordinates and the calculated pixel coordinates, so as to improve the measurement accuracy of the instrument; Step three, a mathematical model for extracting the two-dimensional image displacement of the railway track fastener is established; the concept of circulant matrix is introduced, the surrounding area of the track fastener is densely sampled to generate a fastener sample set; the fastener features are mapped to a high-dimensional space by using the Gaussian kernel function to make them linearly separable; the maximum value of the response function is calculated, which is the region of the fastener in the next frame, so as to complete the extraction of the two-dimensional image displacement of the railway track fastener; the corresponding relationship between the two-dimensional image and the three-dimensional real coordinate established in steps one and two is used to convert the two-dimensional image displacement of the fastener into the three-dimensional real space displacement of the fastener; Step four, through laboratory equipment to simulate various working conditions in the construction process, use multiple cameras in different directions to monitor the track for a long time, use high-definition cameras equipped with zoom lenses and multiple sizes of high-precision targets to collect data, and use the method of steps one to three to extract the spatial displacement of the track fastener.

2. The method of claim 1, wherein The step one specifically comprises the following steps: Step one, the coordinate of each mark point i on the plane target in the world coordinate system is represented as (U i V i 1) T , and the coordinate of each mark point i in the pixel coordinate system is represented as (u i v i 1) T , A is the camera intrinsic matrix, the first two columns of the extrinsic matrix P are denoted as R1 and R2, the last column is denoted as T1, and z i is the scaling coefficient, and formula (1) is obtained: Let A(R1R2T1) be denoted as homography matrix H, and let the nine elements of matrix H be denoted as H 11 -H 33 The first three columns of H are denoted as (H1, H2, H3), and substituting equation (1) gives: Step one two, elimination of z from formula (2) i yields: In the formula, H is a non-full rank homogeneous matrix, R1 and R2 satisfy the unit orthogonal relationship, then: R1 T R2= 0 (4) R1 T R1= R2 T R2= 1 (5) According to the relationship between H and R, it can be known that: R1= A -1 H1, R2= A -1 H2 (6) Step one three, substituting equation (6) into (4) and (5) can obtain: H1 T A -T A -1 H1= H2 T A -T A -1 H2= 1 (8) Let A -T A -1 = B, the nine elements of B are denoted as B 11 -B 33 , α, β and γ are three equivalent parameters, and u0 and v0 are the coordinates of the center point of the imaging plane in the pixel coordinate system, and the following can be obtained: From equation (9), it can be known that B is a symmetric matrix, and the six-dimensional vector is represented as: b = [B 11 B 12 B 22 B 13 B 23 B 33 ] T (11) In the formula Step one four, the basic constraint equation of the camera internal parameter can be rewritten as: The camera internal parameter can be represented as: According to A(R1 R2 T)=H, the (R1 R2 T) corresponding to each sample is obtained, according to the spatial orientation relationship of R1, R2 and R3, it can be obtained that: R3=R1×R2 (16) So as to complete the calculation of the internal and external parameters of the lens.

3. The method of claim 1, wherein The step two specifically comprises the following steps: Step two one, calibrate the lens distortion, according to Zhang Zhengyou calibration method, consider the radial distortion which has greater influence in the distortion model: where (x, y) and (u, v) are the normalized image coordinates without and with distortion, respectively, and r is the distance of a pixel point to the center point of the image, i.e., r = x2+ y2. 2 2 2 where k1and k2are distortion coefficients, and the coordinate transformation relationship between the coordinates (u, v) in the image coordinate system and the coordinates in the pixel coordinate system is:​​​ Step two, distortion image pixel coordinate system Expression: Substitute the radial distortion formula, then: Simplify it: Expressed in matrix form: Let the left half of the left side of the above equation be D, and the right half be k, and simplify the expression: Dk=d (23) Solve the radial distortion coefficient by least square: Step two three, calculate the re-projection error by the actual pixel coordinates and the calculated pixel coordinates, minimize the re-projection error, and repeatedly iterate the internal and external parameters of the camera and the distortion coefficient until convergence.

4. The method of claim 1, wherein The step three specifically comprises the following steps: Step three one, a mathematical model of railway track fastener two-dimensional image displacement extraction is established, that is, a filter is designed, when the filter acts on the fastener area, the position of the maximum response is the position of the fastener, and the filter is represented by the following formula: In the formula, g represents the correlation output, f represents the input image, and h represents the filter template; according to the convolution theorem in Fourier transform, the following form is obtained in the frequency domain: G = F H * (28) In the formula, G, F and H correspond to the Fourier transform of g, f and h respectively; The tth frame is taken as a training set for training, to obtain the correlation filter corresponding to the current frame, and then the correlation operation is performed on the t+1th frame image, to obtain the maximum response, and then the inverse Fourier transform is performed, to obtain the predicted position of the fastener in the t+1th frame; In the training stage, the relevant filter is trained by ridge regression, and let x represent the input of the fastener image i , the trained filter weight is ω i , and the output can be expressed as: f(x i ) = ω T x i (29) At this time, the tracking problem is converted into a ridge regression problem in machine learning, and the objective function is: where y i is the label of the training sample in the training set, λ is the coefficient of the penalty term to avoid overfitting, and formula (30) has a closed-form solution. ω = (X H X + λI) -1 X H y (31) wherein X is the conjugate transpose matrix of X H is the conjugate transpose matrix of X Step three two, in order to reduce the calculation amount, the concept of circulant matrix is introduced, the column vectors in all the sample sets after displacement are arranged by rows, that is, the circulant matrix X is obtained, as shown in formula (34): According to the properties of the circulant matrix, diagonalization is performed through discrete Fourier transform, and the expression is: where is the discrete Fourier transform of the vector x, F is the Fourier transform matrix of order corresponding; substituting equation (35) into equation (31), the inversion of the circulant matrix is equivalent to the inversion of the eigenvalues, using the above property we get: wherein represents a convolution operation; using the circulant matrix convolution property, one can obtain the expression of equation (36) in the frequency domain: In the formula, represents dot multiplication; Step three, introduce kernel function, through nonlinear mapping to high-dimensional feature space, so as to achieve linear separable, let: Formula (29) can be rewritten as: The problem is transformed into finding the optimal solution of each α i The optimal solution is found by taking the derivative, which, in matrix form, is written as a * = (K + λI) -1 y (40) In the formula, α * is the optimal solution matrix of α, K is the kernel matrix composed of sample points {K ij |K ij = k(x i , x j )}, and y is the matrix composed of each y i ; According to the above, the optimal solution in the frequency domain can be obtained: Step three four, in the detection stage, after obtaining the filter of the current frame through training, the position of the fastener in the current frame is taken as the center in the image of the next frame, and dense sampling is performed in the sliding window traversal mode to form a sample set z; The filter of the current frame is used to process the calculation sample z, and the result is as follows: where a is the filter trained from the current frame, k(z, x i ) denotes the kernel function, which is transferred to the frequency domain by Fourier transform, and has: In the formula, K xz is the nuclear matrix K z = k(z i ,x i ) of the reference row vector, and finally the detection area corresponding to the maximum of the response function is the fastener area in the next frame. After each frame of image is detected, in order to adapt to the appearance change of the fastener object in the subsequent tracking process, the filter template is updated, and the update formula is determined through formula (44) and formula (45): x t = (1 - η)x t-1 + ηx' (44) α t = (1 - η)α t-1 + ηα' x (45) In the above updating formula, x' represents the current detected sample, x t and x t-1 represent the fastener sample of the current frame and the previous frame respectively, a' x represents the coefficient obtained by training when the input sample is x', a t and a t-1 represent the coefficients when the response function is maximum in the current frame and the previous frame respectively, and η is the learning rate. Step three five, the corresponding relationship between two-dimensional image and three-dimensional real coordinate established by steps one and two is used to convert the two-dimensional image displacement of the fastener into the three-dimensional real space displacement of the fastener.

Images