Real-time track gauge measuring method based on track inspection vehicle camera and machine vision algorithm

Abstract

The invention discloses a real-time track gauge measurement method based on a track inspection vehicle camera and a machine vision algorithm, and the method comprises the following steps: shooting and collecting a track structure picture, and obtaining an image data set; constructing a semantic segmentation model based on a U-Net neural network; carrying out steel rail marking and image enhancement on the obtained image data set, training the constructed semantic segmentation model, and obtaining a steel rail image data set through the trained semantic segmentation model; in the obtained steel rail image data set, steel rail tread labeling and image enhancement are carried out, and after a constructed semantic segmentation model is trained, a steel rail tread image data set is obtained through the trained semantic segmentation model; carrying out rough search on the obtained steel rail tread image data set to obtain a steel rail tread center line subjected to rough search; performing sub-pixel-level detection on the obtained roughly searched steel rail tread center line, and determining a sub-pixel-level center line; after the sub-pixel-level center line is determined, the rail gauge of the real world is obtained through calculation. According to the invention, by introducing semantic segmentation of the U-Net neural network and a sub-pixel-level center line detection algorithm, the track gauge measurement precision is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of railway track structure safety, and specifically relates to a real-time gauge measurement method based on a track inspection vehicle camera and machine vision algorithms. Background Art

[0002] In recent years, with the improvement of the informatization and automation levels in the railway industry, railway track structure safety detection technologies have also developed significantly. As an important part of track parameters, the accuracy of the gauge directly affects the safety and stability of train operation.

[0003] Traditional gauge measurement methods mainly rely on mechanical measuring tools or fixed sensor systems. Mechanical measuring tools rely on physical contact for measurement, which may cause tool wear and is not suitable for high-speed moving or continuous monitoring scenarios; fixed sensors need to be installed on the track, and the layout and maintenance costs in different track sections are relatively high, with poor flexibility and difficulty in meeting the diverse needs of the railway network.

[0004] With the rapid development of machine vision and artificial intelligence technologies, introducing them into gauge measurement has become an important direction to solve the above problems. Through high-resolution cameras and advanced image processing algorithms, not only can non-contact measurement be achieved, but also the data processing efficiency and measurement accuracy can be improved.

[0005] Existing technologies still have certain limitations in terms of real-time performance, intelligence, and measurement accuracy. Especially in aspects such as how to efficiently collect track images in combination with track inspection vehicles, how to accurately extract rail features through semantic segmentation algorithms, and how to use centerline detection technologies to achieve high-precision gauge calculation, there is still a lack of systematic solutions. Summary of the Invention

[0006] The purpose of the present invention is to provide a real-time gauge measurement method based on a track inspection vehicle camera and machine vision algorithms for calculating the gauge with high precision and high accuracy.

[0007] The real-time gauge measurement method based on a track inspection vehicle camera and machine vision algorithms provided by the present invention includes the following steps:

[0008] S1. Photograph and collect pictures of the track structure to obtain an image data set;

[0009] S2. Build a semantic segmentation model based on the U-Net neural network;

[0010] S3. Perform rail annotation and image enhancement on the image data set obtained in step S1, train the constructed semantic segmentation model, and obtain a rail image data set through the trained semantic segmentation model;

[0011] S4. In the rail image dataset obtained in step S3, perform the annotation and image enhancement of the rail tread. After training the constructed semantic segmentation model, use the trained semantic segmentation model to obtain the rail tread image dataset;

[0012] S5. Conduct a rough search on the rail tread image dataset obtained in step S4 to obtain the roughly searched center line of the rail tread;

[0013] S6. Perform sub-pixel level detection on the roughly searched center line of the rail tread obtained in step S5 to determine the sub-pixel level center line;

[0014] S7. After determining the sub-pixel level center line, calculate the rail gauge in the real world.

[0015] Step S1 is specifically as follows:

[0016] Run the integrated railway rail defect detection vehicle, and synchronously trigger the line array camera mounted on the vehicle through the encoder. Use the line array camera to take top-down pictures of the track structure. These pictures consist of the left rail, the right rail, and the area between the two rails; the encoder triggering scheme is specifically as follows: Set the camera to respond to the encoder signal to ensure that the camera triggers image acquisition when receiving the encoder signal; when the camera receives the encoder signal, each line array camera synchronously triggers to acquire images; the acquired track structure pictures include various rail pictures with different styles, covering special situations such as foreign object intrusion and turnout rails.

[0017] Step S2 is specifically as follows:

[0018] The semantic segmentation model is based on the U-Net neural network; the U-Net neural network includes an encoding part, a decoding part, and a prediction part; the encoding part includes a first encoder, a second encoder, a third encoder, a fourth encoder, and a fifth encoder connected in series in sequence; the first encoder and the second encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the third encoder and the fourth encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the fifth encoder includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder are used to extract track image features of different scales from the input track image data; the decoding part includes a first decoder, a second decoder, a third decoder, and a fourth decoder connected in series in sequence; the first decoder, the second decoder, the third decoder, and the fourth decoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the output of the fifth encoder is upsampled by 2×2 and then concatenated with the output of the fourth encoder to obtain a first fusion feature as the input of the first decoder; the output of the first decoder is upsampled by 2×2 and then concatenated with the output of the third encoder to obtain a second fusion feature as the input of the second decoder; the output of the second decoder is upsampled by 2×2 and then concatenated with the output of the second encoder to obtain a third fusion feature as the input of the third decoder; the output of the third decoder is upsampled by 2×2 and then concatenated with the output of the first encoder to obtain a fourth fusion feature as the input of the fourth decoder; the decoding part gradually restores the features of the encoding part to the original image size through upsampling and performs concatenation; the prediction part includes a 1×1 convolutional layer; the output of the fourth decoder is used as the input of the prediction part; the prediction part is used to convert the final fusion feature layer into a prediction result.

[0019] Step S3 is specifically as follows:

[0020] Before annotating the image dataset obtained in step S1, the steel rail and the switch guard rail are divided into two categories, and Labelme is used to annotate the steel rail images. The annotation includes the entire steel rail of the rail tread, the rail web, and the rail base; for the annotated images, OpenCV is used to randomly scale, translate, mirror flip, transform colors, and add noise to the images to enhance the image dataset to obtain better model performance; during the training process, cross-entropy loss and Dice loss are used as loss functions, and by minimizing the combination of these two loss functions, the model prediction result is made closer to the true label. The combined loss function is:

[0021] L = α·L CE +(1 - α)·L Dice

[0022] where: α is a weighting term used to balance the contribution ratio between the cross - entropy loss and the Dice loss; L CE is the cross - entropy loss, and L Dice is the Dice loss;

[0023] The cross - entropy loss L CE is:

[0024]

[0025] where: N is the number of samples, y i is the true label of the i - th sample, is the label predicted by the model;

[0026] The Dice loss L Dice is:

[0027]

[0028] where: p i is the predicted pixel probability of the i - th sample by the model, t i is the binary pixel value of the true label of the i - th sample, and ε is a smoothing term.

[0029] Step S5 is specifically as follows:

[0030] a. Establish an image subset on the images of the rail tread image dataset obtained in step S4;

[0031] b. Perform a vertical subset search on the image subset obtained in step a and the images of the rail tread image dataset to obtain a candidate subset;

[0032] c. Perform a line search on the candidate subset obtained in step b to obtain the center line of the rough search.

[0033] Step a is specifically as follows:

[0034] Designate the upper - left corner of the image of the rail tread image dataset obtained in step S4 as the origin of the image coordinate system, construct a square region in the image as the image subset, and use f=(x, y) to represent the intensity value of the search point, where x is the abscissa and y is the ordinate.

[0035] Step b is specifically as follows:

[0036] Perform a vertical subset search on the image subset obtained in step a; place the image subset at the top of the left and right sides of the image respectively, and slide the image subset from top to bottom to find the maximum intensity g e at. If the maximum intensity g e is less than the intensity threshold G et, then use the position of this subset as a candidate for the rail endpoint; define the maximum intensity g here e is the maximum value of the pixel gray values within the current search subset during the rough centerline search process; the intensity threshold G et is the average value of the gray values of the background pixels; if no qualified candidate subset is found after a round of vertical search, then move the image subset parallelly and perform a new round of vertical search until at least one qualified candidate subset appears at both the left and right ends of the image.

[0037] Step c is specifically as follows:

[0038] Define the maximum intensity g l is the maximum value of the pixel gray values within the current search subset, and the intensity threshold G el is the average value of the gray values of the target centerline pixels; obtain the intensity threshold G el When doing so, first conduct a rough line search throughout the image, use the gradient-based method to find the possible centerline pixels, and record the positions and gray values of these pixels; use the centerline pixels obtained from the preliminary search to calculate the average value of the gray values of these pixels as the initial intensity threshold G el ; assume that N centerline pixels are obtained from the preliminary search, and the gray value of each pixel is I i , then the initial intensity threshold G el is calculated as follows:

[0039]

[0040] Then conduct the centerline search. Conduct the line search by constructing a search window along the line. The search window is smaller than the rail width; use the maximum intensity g l as the target intensity value, compare the maximum intensity g l with the intensity threshold G el , if g l <G el , then this line is regarded as an accurate estimate of the centerline position and endpoint coordinates, and the centerline of the rough search is obtained; during the centerline search process, if g l >G el , then recalculate G el ; use the average value of all the pixel gray values within the current search window to update G el ; assume that the pixel gray values within the current search window are I1, I2, I3 ··· I M , then the updated G el is:

[0041]

[0042] After updating the intensity threshold G el , conduct the centerline search again to determine the centerline of the rough search.

[0043] Step S6 is specifically as follows:

[0044] First, determine the endpoint positions of the centerline. Then, taking these endpoints as centers, find the upper and lower limits of the line and construct a sub-pixel search subset. The sub-unit set of sub-pixels refers to a set of finer pixel points generated by interpolation between the original image pixels, which is used to more precisely find the top and bottom edges of the centerline. Around the preliminarily determined endpoints, construct a search subset whose size is smaller than the diameter of the centerline, and then use bicubic interpolation to interpolate the selected subset. By setting the resolution, slide the sub-pixel search subset in the vertical and horizontal directions to search for the top and bottom edges of the centerline. If the maximum intensity g e is less than the intensity threshold G et , then the maximum intensity g e , that is, the pixel point with the largest gray value, then this subset is regarded as being completely covered by a straight line, and the centerline endpoints of this subset are selected as candidate nodes. When there are more than one candidate nodes, the continuous and adjacent candidate nodes are regarded as a region, and the center point of this region is selected as a single candidate node. Connect the candidate nodes of the endpoints, and then the sub-pixel level centerline is obtained.

[0045] Step S7 is specifically as follows:

[0046] (1) Calibrate the internal and external parameters of the camera to determine the calibration parameters of the camera;

[0047] (2) Select the centerline of the left rail in the sub-pixel level centerline obtained in step S6, mark any point on it as point C and draw a perpendicular line. The intersection point of the perpendicular line and the centerline of the right rail is marked as D. After obtaining the pixel coordinates of point C and point D, solve the world coordinates of point C and point D through the calibration parameters;

[0048] (3) Calculate the rail gauge through the world coordinates of point C and point D;

[0049] Step (1) is specifically as follows:

[0050] The calibration process is as follows:

[0051] Let the two-dimensional point m=(u, v) in the pixel coordinate system T , and the three-dimensional point in the world coordinate system is represented by M=(X, Y, Z) T , and its augmented matrix is and where u represents the abscissa in the pixel coordinate system, v represents the ordinate in the pixel coordinate system, and (u, v) is the pixel position in the image; X represents the abscissa in the world coordinate system, Y represents the ordinate in the world coordinate system, Z represents the height coordinate in the world coordinate system, and (X, Y, Z) is the position of the point in the actual three-dimensional space;

[0052] Therefore, the relationship between the three points and their projected image points is as follows:

[0053]

[0054] In the formula: s is an arbitrary standard vector, the A matrix is the camera internal parameter, the rotation matrix R and the translation vector t are external parameters;

[0055] The rotation matrix R is expressed as:

[0056]

[0057] In the formula: α is the rotation angle around the Z axis; β is the rotation angle around the Y axis; γ is the rotation angle around the X axis;

[0058] The internal parameter matrix A is expressed as:

[0059]

[0060] In the formula: (u0, v0) are the principal point coordinates of the camera in the pixel coordinate system, a and b are the scale factors of the u and v coordinate axes on the image, and c represents the perpendicularity of the pixel coordinate axes;

[0061] Assuming that the template plane is on the plane of Z = 0 in the world coordinate system, then there is:

[0062]

[0063] In the formula: r1 represents the first column of the rotation matrix, r2 represents the second column of the rotation matrix, r3 represents the third column of the rotation matrix, and r1, r2, r3 are used to describe the rotation part from the world coordinate system to the pixel coordinate system; t represents the translation vector, which is used to describe the translation part from the world coordinate system to the pixel coordinate system;

[0064] The homogeneous coordinates on the checkerboard plane And are the homogeneous coordinates of the points on the checkerboard plane projected onto the corresponding points on the camera imaging plane;

[0065] Therefore, a 3×3 matrix H = (h1 h2 h3) = λA(r1 r2 t) is obtained. The constraint condition for obtaining the internal parameter matrix A using the homography matrix is In the formula: h1 is the first column of the homography matrix H, representing the projection of the X-axis direction in the world coordinate system in the pixel coordinate system; h2 is the second column of the homography matrix H, representing the projection of the Y-axis direction in the world coordinate system in the pixel coordinate system; h3 is the third column of the homography matrix H, representing the translation part of the origin of the world coordinate system in the pixel coordinate system;

[0066] Therefore, the external parameter rotation matrix R and translation vector t of the camera are solved using the following formula:

[0067]

[0068] Where: λ is a scalar used to normalize the lengths of the rotation vectors r1 and r2 to make them unit vectors;

[0069] Step (2) is specifically as follows:

[0070] Find the line spacing calculation points C and D, whose pixel coordinates are defined as follows: The pixel coordinates of points C and D are (u1, v1) and (u2, v2) respectively, where u represents the abscissa in the image and v represents the ordinate in the image;

[0071] Convert the pixel coordinates to world coordinates through the internal and external parameter matrices of the camera, and the operation is as follows: First, calibrate the external parameters of the camera using the Zhang Zhengyou calibration method to determine the rotation matrix R and translation matrix t of the camera. Then, according to the internal and external parameter matrices of the camera, convert the pixel coordinates (u1, v1) of point C in the image to world coordinates (X1, Y1). Similarly, convert the pixel coordinates (u2, v2) of point D to (X2, Y2);

[0072] Therefore, the pixel coordinates of points C and D are C(u1, v1) and D(u2, v2) respectively. Then, substitute into the following formula according to the above calibration parameters to solve the world coordinates C(X1, Y1) and D(X2, Y2):

[0073]

[0074] Convert the image coordinates to camera coordinates:

[0075]

[0076] Where: f is the focal length of the camera; X c 、Y c 、Z c are the coordinates in the camera coordinate system;

[0077] Then, convert to the world coordinate system by the external parameter matrix:

[0078]

[0079] Where: X c 、Y c 、Z c are the coordinates in the camera coordinate system; R is the rotation matrix, which is a 3×3 orthogonal matrix used to describe the rotation from the world coordinate system to the camera coordinate system; X w 、Y w 、Z ware coordinates in the world coordinate system; T is the translation vector, which is a 3×1 vector representing the position of the origin of the world coordinate system in the camera coordinate system;

[0080] In summary, through the conversion of the above four coordinate systems, points C and D are converted from the pixel coordinate system to the world coordinate system, and the conversion formula is as follows:

[0081]

[0082] In the formula: f is the camera focal length; X v , Y v , Z v are coordinates in the world coordinate system, with the subscript v indicating points in the world coordinate system; f x is the component of the camera focal length in the x direction, and f y is the component of the camera focal length in the y direction;

[0083] Step (3) is specifically as follows:

[0084] From the two-point distance formula the distance between the center lines is obtained;

[0085] Since the gauge is the minimum distance between the working edges of the two rails within 16 mm below the rail tread (top surface), the 16 mm measurement point is defined here as the gauge measurement point;

[0086] The distance d0 from the axis of symmetry of the rail to the 16 mm measurement point in the rail profile is calculated from the rail dimension diagram, so the rail gauge is l = d - d0.

[0087] The beneficial effects of the present invention are as follows:

[0088] 1. By introducing the semantic segmentation and sub-pixel level center line detection algorithms of the U-Net neural network, the present invention greatly improves the accuracy of gauge measurement. At the same time, combined with the high-resolution linear array camera carried by the track inspection vehicle, it realizes the real-time dynamic monitoring of the gauge, meeting the high-precision measurement requirements for high-speed railway operation;

[0089] 2. Using machine vision and deep learning algorithms, the present invention can adapt to complex backgrounds and changing lighting conditions, effectively reducing the interference of the environment on the measurement accuracy; at the same time, it realizes a high degree of automation of gauge detection, reduces human intervention, and improves the efficiency of railway track safety detection. Description of the Drawings

[0090] Figure 1 is a schematic flow chart of the method of the present invention. Detailed Embodiments

[0091] As Figure 1As shown in the figure, the present invention provides a real-time gauge measurement method based on a track inspection vehicle camera and a machine vision algorithm, comprising the following steps:

[0092] S1. Take pictures of the track structure to obtain an image dataset, specifically:

[0093] Run a railway steel rail comprehensive defect detection vehicle, and synchronously trigger three line array cameras mounted on the vehicle through an encoder. The three line scan cameras are used to take top-down pictures of the track structure. These pictures mainly consist of the left rail, the right rail, and the area between the two rails.

[0094] Since the line array camera has a high resolution and can capture more detailed fine defects on the steel rail surface, and the line array camera uses a progressive scanning method for continuous image acquisition, the steel rail detection vehicle can provide a stable and fast data acquisition environment.

[0095] To ensure the continuity of the images collected by the three cameras, the encoder trigger scheme is specifically as follows:

[0096] Set the camera in the camera configuration interface to respond to the encoder signal to ensure that the camera triggers image acquisition when receiving the encoder signal; then write code to process the encoder signal. Once the encoder signal reaches the trigger condition, the camera starts to collect images, and finally the three line array cameras are synchronously triggered to collect images.

[0097] The present invention verifies that the three cameras can be stably synchronously triggered under different conditions through image stitching, meeting the requirements of image acquisition.

[0098] The collected track structure pictures include various steel rail pictures with different styles, covering special situations such as foreign object intrusion and turnout steel rails.

[0099] S2. Build a semantic segmentation model based on the U-Net neural network, specifically including the following steps:

[0100] The semantic segmentation model is based on the U-Net neural network;

[0101] The U-Net neural network includes an encoding part, a decoding part, and a prediction part;

[0102] The encoding part includes a first encoder, a second encoder, a third encoder, a fourth encoder, and a fifth encoder connected in series in sequence; the first encoder and the second encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the third encoder and the fourth encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the fifth encoder includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder are used to extract track image features of different scales from the input track image data;

[0103] The decoding part includes a first decoder, a second decoder, a third decoder, and a fourth decoder connected in series in sequence; the first decoder, the second decoder, the third decoder, and the fourth decoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the output of the fifth encoder is upsampled by 2×2 and then concatenated with the output of the fourth encoder to obtain a first fusion feature as the input of the first decoder; the output of the first decoder is upsampled by 2×2 and then concatenated with the output of the third encoder to obtain a second fusion feature as the input of the second decoder; the output of the second decoder is upsampled by 2×2 and then concatenated with the output of the second encoder to obtain a third fusion feature as the input of the third decoder; the output of the third decoder is upsampled by 2×2 and then concatenated with the output of the first encoder to obtain a fourth fusion feature as the input of the fourth decoder; the decoding part gradually restores the features of the encoding part to the original image size through upsampling and performs concatenation;

[0104] The prediction part includes a 1×1 convolutional layer; the output of the fourth decoder is used as the input of the prediction part; the prediction part is used to convert the final fusion feature layer into a prediction result.

[0105] S3. After performing rail annotation and image enhancement on the image dataset obtained in step S1 and training the constructed semantic segmentation model, obtain the rail image dataset through the trained semantic segmentation model, which specifically includes the following steps:

[0106] Before annotating the image dataset obtained in step S1, classify the rails and switch guards into two categories, and use Labelme to annotate the rail images. The annotation includes the entire rail of the rail tread, rail web, and rail base; for the annotated images, use OpenCV to randomly scale, translate, mirror flip, perform color transformation, add noise, etc. on the images to enhance the image dataset to obtain better model performance;

[0107] During the training process, the loss function is used as the optimization objective. By minimizing the loss function, the model can learn how to make the prediction results closer to the true labels, thereby improving the prediction accuracy and generalization ability. Specifically, the cross-entropy loss and the Dice loss are used as the optimization objectives. By minimizing the combination of these two loss functions, the model prediction results are made closer to the true labels. The combined loss function is defined as:

[0108] L = α·L CE +(1 - α)·L Dice

[0109] Where: α is the weighting term, used to balance the contribution ratio between the cross-entropy loss and the Dice loss; L CE is the cross-entropy loss, and L Dice is the Dice loss.

[0110] The cross-entropy loss is usually used to measure the difference between the model output and the true labels. For binary classification problems, the cross-entropy loss can be used as the loss function. The cross-entropy loss is a commonly used loss function in machine learning and deep learning. It measures the difference between two probability distributions and is often used to measure the accuracy of model predictions. The cross-entropy loss L CE is:

[0111]

[0112] Where: N is the number of samples, y i is the true label of the i-th sample, is the label predicted by the model.

[0113] The Dice loss is a loss function used for image segmentation tasks. It measures the similarity between the predicted segmentation result of the model and the true segmentation.

[0114] The Dice coefficient is defined as the ratio of the intersection of two geometries to their average size. The Dice loss L Dice is expressed as:

[0115]

[0116] Where: p is the pixel probability predicted by the model, t is the binary pixel value of the true label, and ε is a smoothing term to avoid the denominator being zero.

[0117] S4. In the rail image dataset obtained in step S3, perform the annotation and image enhancement of the rail tread. After training the constructed semantic segmentation model, obtain the rail tread image dataset through the trained semantic segmentation model. The specific training process is the same as that in step S3.

[0118] S5. Coarsely search the rail tread image dataset obtained in step S4 to obtain the coarsely searched center line of the rail tread. The specific steps are as follows:

[0119] a. Establish an image subset on the images of the rail tread image dataset obtained in step S4.

[0120] b. Conduct a vertical subset search on the image subset obtained in step a and the images of the rail tread image dataset to obtain qualified candidate subsets.

[0121] c. Conduct a line search on the candidate subsets obtained in step b to obtain the coarsely searched center line.

[0122] Step a is specifically as follows:

[0123] Designate the upper left corner of the image of the rail tread image dataset obtained in step S4 as the origin of the image coordinate system, construct a square area in the image as the image subset, and use f = (x, y) to represent the intensity value of the search point, where x is the abscissa and y is the ordinate.

[0124] Step b is specifically as follows:

[0125] To locate potential rail endpoints in the image, conduct a vertical subset search on the image subset obtained in step a.

[0126] Place the image subset at the top of the left and right sides of the image respectively, and slide the image subset from top to bottom to find the maximum intensity g e at the position. If the maximum intensity g e is less than the intensity threshold G et , then take the position of this subset as a candidate object for the rail endpoint. Here, define the maximum intensity g e as the maximum value of the pixel gray values within the current search subset (sub-unit set) during the coarse center line search process; the intensity threshold G et is the average value of the background pixel gray values.

[0127] If no qualified candidate subset is found after a round of vertical search, move the image subset parallelly and perform a new round of vertical search until at least one qualified candidate subset appears at both ends of the image.

[0128] Step c is specifically as follows:

[0129] Multiple qualified candidate subsets may appear in the vertical subset search in step b. At this time, multiple straight lines will appear when connecting the left and right candidate nodes, but the center line of the rail is uniquely determined. Therefore, it is necessary to find a relatively accurate position of the rail center line.

[0130] Define the maximum intensity g l as the maximum value of the pixel gray values within the current search subset, and the intensity threshold Gel is the average gray value of the target center line pixels;

[0131] Obtain the intensity threshold G el When doing this, first perform a rough line search in the entire image, use the gradient-based method to find the possible center line pixels, and record the positions and gray values of these pixels; use the center line pixels obtained from the preliminary search to calculate the average value of the gray values of these pixels as the initial intensity threshold G el ;

[0132] Assume that N center line pixels are obtained from the preliminary search, and the gray value of each pixel is I i , then the initial intensity threshold G el is calculated as follows:

[0133]

[0134] Then perform a center line search. Conduct a line search by constructing a search window along the line. The search window is smaller than the rail width; take the maximum intensity g l as the target intensity value, compare the maximum intensity g l with the intensity threshold G el . If g l <G el , then this line is regarded as an accurate estimate of the center line position and endpoint coordinates, and the center line obtained from the rough search is obtained;

[0135] During the center line search process, if g l >G el , then recalculate G el ; use the average value of all pixel gray values within the current search window to update G el ; assume that the pixel gray values within the current search window are I1, I2, I3 ··· I M , then the updated G el is:

[0136]

[0137] Update the intensity threshold G el After that, perform a center line search again to determine the center line obtained from the rough search.

[0138] S6. Perform sub-pixel level detection on the center line of the rail tread obtained from the rough search in step S5 to determine the center line at the sub-pixel level. The specific steps are as follows:

[0139] After the rough search for the center line, the candidate subsets at the left and right ends of the image can obtain a rough estimate of the center line. However, the accuracy at this time is at the whole pixel level and cannot meet the actual engineering requirements. Therefore, sub-pixel accuracy identification is required.

[0140] Perform sub-pixel level detection on the center line searched by S6. First, determine the endpoint positions of the center line, and then, with these endpoints as the centers, find the upper and lower limits of the line to construct a sub-pixel search subset. The sub-unit set of sub-pixels refers to a set of finer pixel points generated by interpolation between the original image pixels, which is used to more precisely find the top and bottom edges of the center line;

[0141] Around the preliminarily determined endpoints, construct a search subset whose size is smaller than the diameter of the center line; then use bicubic interpolation to interpolate the selected subset, and by setting a vertical step of 0.5 pixels and a resolution of 0.5 pixels, slide the sub-pixel search subset in the vertical and horizontal directions to search for the top and bottom edges of the center line.

[0142] Bicubic interpolation is used to determine the sub-pixel intensity values, and its main goal is to accurately determine the center point of the target area by using interpolation to approach the upper and lower limits of the line.

[0143] Similar to the intensity condition of the rough search, if the maximum intensity g e is less than the intensity threshold G et , and the maximum intensity g e , that is, the pixel point with the largest gray value, then this subset is considered to be completely covered by a straight line, and the center line endpoints of this subset are selected as candidate nodes.

[0144] When there are multiple candidate nodes, the continuous and adjacent candidate nodes are regarded as a region, and the center point of this region is selected as a single candidate node.

[0145] Connect the candidate nodes of the endpoints, and then the sub-pixel level center line is obtained.

[0146] S7. After determining the sub-pixel level center line, calculate the real-world rail gauge, and the specific steps are as follows:

[0147] (1) Calibrate the internal and external parameters of the camera to determine the calibration parameters of the camera.

[0148] (2) Select the center line of the left rail among the sub-pixel level center lines obtained in step S6, mark any point on it as point C and draw a perpendicular line. The intersection point of the perpendicular line and the center line of the right rail is marked as D. After obtaining the pixel coordinates of points C and D, solve the world coordinates of points C and D through the calibration parameters;

[0149] (3) Calculate the rail gauge through the world coordinates of points C and D.

[0150] Step (1) is specifically:

[0151] To accurately relate the images captured by the camera to the physical coordinates in the real world, it is necessary to calibrate the internal and external parameters of the camera. The internal parameters mainly describe the characteristics of the camera itself, such as focal length, principal point offset, distortion, etc., while the external parameters describe the position and orientation of the camera in the world coordinate system.

[0152] The calibration process is as follows:

[0153] The calibration board is placed 650 mm away from the camera, and continuous acquisition is carried out by pushing the trolley, and it is ensured that the position of the calibration board is not uniquely fixed. Take 10 - 20 images.

[0154] Let the two - dimensional point m=(u, v) in the pixel coordinate system T , and the three - dimensional point in the world coordinate system can be represented by M=(X, Y, Z) T The augmented matrix is and where u represents the abscissa in the pixel coordinate system, v represents the ordinate in the pixel coordinate system, and (u, v) is the pixel position in the image; X represents the abscissa in the world coordinate system, Y represents the ordinate in the world coordinate system, Z represents the height coordinate in the world coordinate system, and (X, Y, Z) is the position of the point in the actual three - dimensional space;

[0155] Therefore, the relationship between the three - dimensional point and its projected image point is:

[0156]

[0157] In the formula: s is an arbitrary standard vector, the A matrix is the internal parameter of the camera, the rotation matrix R and the translation vector t are the external parameters.

[0158] The rotation matrix R can be expressed as:

[0159]

[0160] In the formula: α is the rotation angle around the Z - axis; β is the rotation angle around the Y - axis; γ is the rotation angle around the X - axis;

[0161] The internal parameter matrix A can be expressed as:

[0162]

[0163] In the formula: (u0, v0) is the principal point coordinate of the camera in the pixel coordinate system, a and b are the scale factors of the u and v coordinate axes on the image, and c represents the perpendicularity of the pixel coordinate axes.

[0164] Assume that the template plane is on the plane of Z = 0 in the world coordinate system, then there is:

[0165]

[0166] Where: r1 represents the first column of the rotation matrix, r2 represents the second column of the rotation matrix, r3 represents the third column of the rotation matrix, and r1, r2, and r3 are used to describe the rotation part from the world coordinate system to the pixel coordinate system; t represents the translation vector, which is used to describe the translation part from the world coordinate system to the pixel coordinate system;

[0167] The homogeneous coordinates on the checkerboard plane while are the homogeneous coordinates of the points on the checkerboard plane projected onto the corresponding points on the camera imaging plane.

[0168] Therefore, a 3×3 matrix H=(h1 h2 h3)=λA(r1 r2 t) can be obtained. Using the homography matrix, the constraint conditions for the internal parameter matrix A are Where: h1 is the first column of the homography matrix H, representing the projection of the X-axis direction in the world coordinate system in the pixel coordinate system; h2 is the second column of the homography matrix H, representing the projection of the Y-axis direction in the world coordinate system in the pixel coordinate system; h3 is the third column of the homography matrix H, representing the translation part of the origin of the world coordinate system in the pixel coordinate system;

[0169] Therefore, the external parameter rotation matrix R and translation vector t of the camera can be solved using the following formula:

[0170]

[0171] Where: λ is a scalar used to normalize the lengths of the rotation vectors r1 and r2 to make them unit vectors;

[0172] Step (2) is specifically as follows:

[0173] Find the line spacing calculation points C and D, whose pixel coordinates are defined as follows: The pixel coordinates of points C and D are (u1, v1) and (u2, v2) respectively, where u represents the abscissa in the image and v represents the ordinate in the image.

[0174] World coordinates refer to the coordinate positions in the actual space. The pixel coordinates are converted into world coordinates through the internal and external parameter matrices of the camera, which are defined as follows: First, use Zhang Zhengyou's calibration method to calibrate the external parameters of the camera to determine the rotation matrix R and translation matrix t of the camera. Then, according to the internal and external parameter matrices of the camera, convert the pixel coordinates (u1, v1) of point C in the image into world coordinates (X1, Y1). Similarly, the pixel coordinates (u2, v2) of point D can be converted into (X2, Y2).

[0175] Therefore, the pixel coordinates of points C and D are C(u1, v1) and D(u2, v2) respectively. Then, solve the world coordinates C(X1, Y1) and D(X2, Y2) according to the above calibration parameters:

[0176]

[0177] Convert the image coordinates to camera coordinates:

[0178]

[0179] Where: f is the camera focal length; X c , Y c , Z c are the coordinates in the camera coordinate system;

[0180] Then, convert it to the world coordinate system by the extrinsic parameter matrix:

[0181]

[0182] Where: X c , Y c , Z c are the coordinates in the camera coordinate system; R is the rotation matrix, which is a 3×3 orthogonal matrix used to describe the rotation from the world coordinate system to the camera coordinate system; X w , Y w , Z w are the coordinates in the world coordinate system; T is the translation vector, which is a 3×1 vector representing the position of the origin of the world coordinate system in the camera coordinate system;

[0183] In summary, through the conversion of the above four coordinate systems, points C and D can be converted from the pixel coordinate system to the world coordinate system, and the conversion formula is as follows:

[0184]

[0185] Where: f is the camera focal length; X v , Y v , Z v are the coordinates in the world coordinate system, with the subscript v indicating the points in the world coordinate system; f x is the component of the camera focal length in the x direction, and f y is the component of the camera focal length in the y direction;

[0186] Step (3) is specifically:

[0187] By the two-point spacing formula the center line spacing can be obtained.

[0188] Since the gauge is the minimum distance between the working edges of the two rails within 16 mm below the rail tread (top surface), the 16 mm measurement point is defined here as the gauge measurement point;

[0189] The distance d0 from the axis of symmetry of the rail in the rail profile to the 16 mm measurement point can be calculated from the rail dimension diagram, so the rail gauge is l = d - d0.

Claims

1. A real-time gauge measurement method based on a track inspection vehicle camera and machine vision algorithm, characterized in that, It includes the following steps: S1. Take pictures of the track structure to obtain an image dataset; S2. Build a semantic segmentation model based on the U-Net neural network; S3. Perform rail annotation and image enhancement on the image dataset obtained in step S1. After training the built semantic segmentation model, obtain a rail image dataset through the trained semantic segmentation model; S4. In the rail image dataset obtained in step S3, perform rail tread annotation and image enhancement. After training the built semantic segmentation model, obtain a rail tread image dataset through the trained semantic segmentation model; S5. Conduct a rough search on the rail tread image dataset obtained in step S4 to obtain a roughly searched center line of the rail tread; S6. Perform sub-pixel level detection on the roughly searched center line of the rail tread obtained in step S5 to determine the sub-pixel level center line; S7. After determining the sub-pixel level center line, calculate the rail gauge in the real world.

2. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 1, wherein, Step S1 is specifically as follows: Run a railway rail comprehensive defect detection vehicle, and synchronously trigger the line array camera carried on the vehicle through an encoder. The line array camera is used to take top-down pictures of the track structure. These pictures consist of the left rail, the right rail, and the area between the two rails; The specific encoder triggering scheme is as follows: Set the camera to respond to the encoder signal to ensure that the camera triggers image acquisition when receiving the encoder signal; when the camera receives the encoder signal, each line array camera synchronously triggers to acquire images; The collected track structure pictures include various rail pictures with different styles, covering foreign object intrusion and turnout rails.

3. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 2, wherein Step S2 is specifically as follows: The semantic segmentation model is based on the U-Net neural network; The U-Net neural network includes an encoding part, a decoding part, and a prediction part; The encoding part includes a first encoder, a second encoder, a third encoder, a fourth encoder, and a fifth encoder connected in series in sequence; the first encoder and the second encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the third encoder and the fourth encoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, and a 2×2 max pooling layer connected in series in sequence; the fifth encoder includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder are used to extract track image features at different scales from the input track image data; The decoding part includes a first decoder, a second decoder, a third decoder, and a fourth decoder connected in series in sequence; the first decoder, the second decoder, the third decoder, and the fourth decoder include a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a third 3×3 convolutional layer connected in series in sequence; the output of the fifth encoder is upsampled by 2×2 and then concatenated with the output of the fourth encoder to obtain a first fusion feature as the input of the first decoder; the output of the first decoder is upsampled by 2×2 and then concatenated with the output of the third encoder to obtain a second fusion feature as the input of the second decoder; the output of the second decoder is upsampled by 2×2 and then concatenated with the output of the second encoder to obtain a third fusion feature as the input of the third decoder; the output of the third decoder is upsampled by 2×2 and then concatenated with the output of the first encoder to obtain a fourth fusion feature as the input of the fourth decoder; the decoding part gradually restores the features of the encoding part to the original image size through upsampling and performs concatenation; The prediction part includes a 1×1 convolutional layer; the output of the fourth decoder is used as the input of the prediction part; the prediction part is used to convert the final fusion feature layer into a prediction result.

4. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 3, characterized in that Step S3 is specifically as follows: Before annotating the image dataset obtained in Step S1, the steel rail and the switch guard rail are divided into two categories, and Labelme is used to annotate the steel rail images, and the annotation includes the entire steel rail of the rail tread, the rail web, and the rail base; For the annotated images, OpenCV is used to randomly scale, translate, mirror flip, perform color transformation, and add noise to the images to enhance the image dataset to obtain better model performance; During the training process, cross-entropy loss and Dice loss are used as loss functions, and by minimizing the combination of these two loss functions, the model prediction result is made closer to the true label. The combined loss function is: L = α·L CE +(1 - α)·L Dice where: α is a weighting term used to balance the contribution ratio between the cross-entropy loss and the Dice loss; L CE is the cross-entropy loss, and L Dice is the Dice loss; Cross-entropy loss L CE is as follows: Where: N is the number of samples, and y i is the true label of the i-th sample, and is the label predicted by the model; Dice loss L Dice is as follows: Where: p i is the model predicted pixel probability of the i-th sample, t i is the true label binary pixel value of the i-th sample, and ε is the smoothing term.

5. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 4, wherein Step S5 is specifically as follows: a. Establish an image subset on the images of the steel rail tread image dataset obtained in Step S4; b. Perform a vertical subset search on the image subset obtained in step a and the images of the steel rail tread image dataset to obtain a candidate subset; c. Perform a line search on the candidate subset obtained in step b to obtain the center line of the rough search.

6. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 5, characterized in that Step a is specifically as follows: Designate the upper left corner of the image of the steel rail tread image dataset obtained in Step S4 as the origin of the image coordinate system, construct a square area in the image as the image subset, and use f=(x,y) to represent the intensity value of the search point, where x is the abscissa and y is the ordinate.

7. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 6, characterized in that, Step b is specifically as follows: Perform a vertical subset search on the image subset obtained in step a; place the image subsets at the top of the left and right sides of the image respectively, and slide the image subsets from top to bottom to find the maximum intensity g e At this point, if the maximum intensity g e is less than the intensity threshold G et , then take the position of this subset as a candidate object for the rail end point; here, the maximum intensity g e is defined as the maximum value of the pixel gray levels within the current search subset during the rough center line search process; the intensity threshold G et is the average value of the gray levels of the background pixels; if no qualified candidate subset is found after a round of vertical search, move the image subset parallelly and perform a new round of vertical search until at least one qualified candidate subset appears at both ends of the image.

8. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 7, wherein Step c is specifically as follows: Define the maximum intensity g l as the maximum value of the pixel gray values within the current search subset, and the intensity threshold G el as the average of the pixel gray values of the target center line; Obtain the intensity threshold G el When doing so, first perform a rough line search throughout the entire image. Use a gradient-based method to find possible centerline pixels and record the positions and gray values of these pixels. Using the centerline pixels obtained from the preliminary search, calculate the average of the gray values of these pixels as the initial intensity threshold G el ; Assume that N centerline pixels are obtained from the preliminary search, and the gray value of each pixel is I i , then the initial intensity threshold G el is calculated as follows: Then perform a centerline search. Conduct a line search by constructing a search window along the line. The search window is smaller than the rail width; with the maximum intensity g l as the target intensity value, take the maximum intensity g l and compare it with the intensity threshold G el If g l <G el , then this line is regarded as an accurate estimate of the centerline position and endpoint coordinates, and the centerline obtained from the rough search is obtained; During the centerline search process, if g l >G el , then recalculate G el ; update G using the average of all pixel grayscale values within the current search window el ; assuming the pixel grayscale values within the current search window are I1, I2, I3 ··· I M , then update G el as follows: Update the intensity threshold G el After that, perform the centerline search again to determine the centerline of the rough search.

9. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 8, characterized in that, Step S6 is specifically as follows: First, determine the endpoint positions of the centerline. Then, taking these endpoints as centers, find the upper and lower limits of the line to construct a sub-pixel search subset. The sub-unit set of sub-pixels refers to a set of finer pixel points generated by interpolation between the original image pixels, which is used to more precisely find the top and bottom edges of the centerline. Around the preliminarily determined endpoints, construct a search subset with a size smaller than the diameter of the centerline, and then use bicubic interpolation to interpolate the selected subset. By setting the resolution, slide the sub-pixel search subset in the vertical and horizontal directions to search for the top and bottom edges of the centerline. If the maximum intensity g e is less than the intensity threshold G et , then the maximum intensity g e , that is, the pixel point with the largest gray value, then this subset is regarded as being completely covered by a straight line, and the centerline endpoints of this subset are selected as candidate nodes. When there are more than one candidate nodes, the consecutive and adjacent candidate nodes are regarded as a region, and the center point of this region is selected as a single candidate node. Connect the candidate nodes of the endpoints, and then the sub-pixel level centerline is obtained.

10. The real-time gauge measurement method based on the track inspection vehicle camera and machine vision algorithm according to claim 9, characterized in that Step S7 is specifically as follows: (1) Calibrate the internal and external parameters of the camera to determine the calibration parameters of the camera; (2) Select the center line of the left rail in the sub-pixel level center lines obtained in Step S6, mark any point on it as point C and draw a perpendicular line, and the intersection point of the perpendicular line and the center line of the right rail is marked as D. After obtaining the pixel coordinates of point C and point D, solve the world coordinates of point C and point D through the calibration parameters; (3) Calculate the rail gauge through the world coordinates of point C and point D; Step (1) is specifically as follows: The calibration process is as follows: Let the two - dimensional point \(m=(u, v)\) in the pixel coordinate system T , and the three - dimensional point in the world coordinate system is represented by \(M=(X, Y, Z)\) T . Its augmented matrix is and where \(u\) represents the abscissa in the pixel coordinate system, \(v\) represents the ordinate in the pixel coordinate system, and \((u, v)\) is the pixel position in the image; \(X\) represents the abscissa in the world coordinate system, \(Y\) represents the ordinate in the world coordinate system, \(Z\) represents the height coordinate in the world coordinate system, and \((X, Y, Z)\) is the position of a point in the actual three - dimensional space Therefore, the relationship between the three points and their projected image points is as follows: In the formula: s is an arbitrary standard vector, the A matrix is the camera internal parameter, the rotation matrix R and the translation vector t are external parameters; The rotation matrix R is expressed as: In the formula: α is the rotation angle around the Z axis; β is the rotation angle around the Y axis; γ is the rotation angle around the X axis; The internal parameter matrix A is expressed as: In the formula: (u0, v0) are the principal point coordinates of the camera in the pixel coordinate system, a and b are the scale factors of the u and v coordinate axes on the image, and c represents the perpendicularity of the pixel coordinate axes; Assume that the template plane is on the plane of Z = 0 in the world coordinate system, then there is: In the formula: r1 represents the first column of the rotation matrix, r2 represents the second column of the rotation matrix, r3 represents the third column of the rotation matrix, and r1, r2, r3 are used to describe the rotation part from the world coordinate system to the pixel coordinate system; t represents the translation vector and is used to describe the translation part from the world coordinate system to the pixel coordinate system; Homogeneous coordinates on the checkerboard plane while are the homogeneous coordinates of the points on the checkerboard plane projected onto the corresponding points on the camera imaging plane; Therefore, a 3×3 matrix H = (h1 h2 h3) = λA(r1 r2 t) is obtained, The constraint condition for obtaining the internal parameter matrix A using the homography matrix is: In the formula: h1 is the first column of the homography matrix H, representing the projection of the X-axis direction in the world coordinate system in the pixel coordinate system; h2 is the second column of the homography matrix H, representing the projection of the Y-axis direction in the world coordinate system in the pixel coordinate system; h3 is the third column of the homography matrix H, representing the translation part of the origin of the world coordinate system in the pixel coordinate system; Therefore, the external parameter rotation matrix R and translation vector t of the camera are solved using the following formula: In the formula: λ is a scalar used to normalize the lengths of the rotation vectors r1 and r2 to make them unit vectors; Step (2) is specifically as follows: Find the line spacing calculation points C and D, and their pixel coordinates are defined as follows: The pixel coordinates of points C and D are (u1, v1) and (u2, v2) respectively, where u represents the abscissa in the image and v represents the ordinate in the image; Convert the pixel coordinates to world coordinates through the internal and external parameter matrices of the camera. The operations are as follows: First, use the Zhang Zhengyou calibration method to calibrate the external parameters of the camera to determine the rotation matrix R and translation matrix t of the camera. Then, according to the internal and external parameter matrices of the camera, convert the pixel coordinates (u1, v1) of point C in the image to world coordinates (X1, Y1). Similarly, convert the pixel coordinates (u2, v2) of point D to (X2, Y2); Therefore, the pixel coordinates of points C and D are C(u1, v1) and D(u2, v2) respectively. Then, substitute into the following formula according to the above calibration parameters to solve the world coordinates C(X1, Y1) and D(X2, Y2): Convert the image coordinates to camera coordinates: where: f is the camera focal length; X c , Y c , Z c are the coordinates in the camera coordinate system; Then convert to the world coordinate system through the external parameter matrix: Where: X c 、Y c 、Z c are the coordinates in the camera coordinate system; R is the rotation matrix, which is a 3×3 orthogonal matrix used to describe the rotation from the world coordinate system to the camera coordinate system; X w 、Y w 、Z w are the coordinates in the world coordinate system; T is the translation vector, which is a 3×1 vector representing the position of the origin of the world coordinate system in the camera coordinate system; In summary, through the conversion of the above four coordinate systems, points C and D are converted from the pixel coordinate system to the world coordinate system, and the conversion formula is as follows: Where: f is the camera focal length; X v , Y v , Z v are the coordinates in the world coordinate system, with the subscript v indicating a point in the world coordinate system; f x is the component of the camera focal length in the x direction, and f y is the component of the camera focal length in the y direction; Step (3) is specifically as follows: From the two-point distance formula the center line spacing is obtained; Since the gauge is the minimum distance between the working edges of the two rails within the range of 16 mm below the rail tread (top surface), a 16-mm measurement point is defined here as the gauge measurement point; The distance d0 from the axis of symmetry of the rail in the rail profile to the 16-mm measurement point is calculated from the rail dimension drawing. Therefore, the rail gauge is l = d - d0.

Images