Image-based three-dimensional railway steel rail center line automatic extraction and reconstruction method and device

Abstract

The invention discloses an image-based three-dimensional railway steel rail center line automatic extraction and reconstruction method and device, and the method comprises the steps: building an RTLP tracking railway line through a simulation train based on a Kalman filtering frame, and carrying out the RTLP initialization and RTLP tracking. Firstly, on the basis of reconstructed scene three-dimensional line segments, RTLP is initialized based on geometric feature constraints and texture alignment based on DBSCAN clustering of depth features, and RTLP seeds are formed. Then, on the basis of the RTLP seeds, predicting the growth direction of the RTLP seeds according to the inertia direction of a train, constructing a predicted value, detecting a two-dimensional railway center line in an image block based on a Hough frame, and performing three-dimensional reconstruction and constructing an observation value based on parallel constraints; and finally, based on a discrete Kalman filter, fusing the observed value and the predicted value, updating the state, and carrying out next Kalman filtering until RTLP is overlapped or the observed value cannot be stably reconstructed. According to the method provided by the invention, the railway center line can be completely reconstructed with high precision.

Description

Technical Field

[0001] The present invention belongs to the method for remote sensing image processing, and particularly relates to a method and device for automatically extracting and reconstructing the center line of a three-dimensional railway rail based on an image. Background Art

[0002] The current railway operating mileage in China has exceeded 1.5 million kilometers. The electrification transformation project of existing railways is gradually becoming one of the key tasks in railway survey and design, and the surveying and mapping work for existing railways will also increase accordingly. One of the core tasks of current railway surveying and mapping is to accurately and efficiently extract the vectorized three-dimensional railway rail line to obtain geometric structure information such as the elevation and width of the railway line, so as to assist in engineering design, monitor the construction quality and ensure the operation safety.

[0003] The vectorized three-dimensional rail line can be obtained from real-time kinematic (RTK), light detection and ranging (LiDAR), and multi-view images. RTK is usually installed on a railway survey vehicle to obtain the rail line by moving along the track. Although the accuracy of the rail line measured based on RTK is very ideal, it needs to occupy the track for operation, affecting the normal railway operation, and there are problems of safety and efficiency. The LiDAR sensor can be installed on an unmanned aerial vehicle (UAV), which is more convenient and safe compared with RTK. However, extracting the rail line based on LiDAR point cloud needs to be achieved by methods such as point cloud segmentation or classification. The UAV needs to maintain a low flight altitude to meet the point cloud density requirements, which will also affect the safety and efficiency. The UAV equipped with an image sensor can efficiently capture high-resolution aerial images at a safe distance from the railway, but there are the following difficulties in extracting the rail line from the aerial images: 1) There are problems in image matching due to occlusion and parallax changes, and the dense point cloud obtained from the aerial images is usually inaccurate near the track; 2) Although the method of combining image semantics is feasible, how to accurately and completely detect and reconstruct the rail line in multi-view aerial images still needs to be studied.

[0004] The core method for rail line extraction based on point clouds is point cloud segmentation. Traditional algorithms guide point cloud segmentation based on designed geometric priors, while learning-based algorithms usually train segmentation models based on PointNet. However, due to inaccurate boundary localization and significant variations in point cloud density, which pose major challenges to robust semantic segmentation, drones need to maintain a low flight altitude to improve point cloud quality and reduce the processing range. Compared with point clouds, images contain rich semantic information that can guide the high-precision extraction of rail lines. Therefore, some studies use deep learning methods to detect rail lines from multi-view images, but more training samples are needed to obtain a more generalizable detection network. Moreover, most of these methods only process single image patches and lack strategies for processing multi-view images. In addition, significant progress has been made in the detection and reconstruction of 3D line segments in recent years, but there is still no algorithm specifically designed for railway track line detection. If the 3D line segments reconstructed directly using these algorithms are used to represent rail lines, problems such as fragmentation and missing segments are likely to occur.

[0005] In summary, there is currently no 3D line segment reconstruction method specifically for railway lines. Whether it is traditional methods or deep learning methods, due to the poor quality of point clouds near railways, it poses a huge challenge to the task of rail line point cloud segmentation. Therefore, the present invention realizes the automatic extraction and reconstruction of the center line of the railway rail top surface based on multi-view high-resolution drone images. Summary of the Invention

[0006] The present invention proposes an automatic method and device for extracting the center line of railway rails based on images to solve the problem of railway center line positioning.

[0007] The technical solution adopted by the present invention is: an automatic method for extracting the center line of railway rails from images, comprising the following steps:

[0008] Step 1, perform image matching on the high-resolution drone images of the input railway scene, and restore the positions and poses of the multi-view images to generate image connection points;

[0009] Step 2, extract the 2D line segments in the images and perform initial 3D line segment reconstruction;

[0010] Step 3, perform geometric alignment on the 3D line segments in Step 2 based on angles, overlap degrees, and projection distances, and construct railway tracking line pair seeds based on DBSCAN clustering of depth features;

[0011] Step 4, predict the center line of the railway rail by simulating the inertial forward direction of the train, and make the railway tracking line pair seeds grow in both side directions;

[0012] Step 5: During the growth process, select the images used to generate the observation line segments of the railway rail centerline based on the number of adjacent three-dimensional connection points; after projecting the growth endpoints of the three-dimensional line segments onto the images, transform the edge features of each local image into the Hough space to extract the two-dimensional railway centerline of the rail.

[0013] Step 6: Perform three-dimensional reconstruction on the multi-view two-dimensional railway centerlines in Step 5 under the constraint of parallel conditions.

[0014] Step 7: Based on the Kalman filter, fuse the predicted railway centerline in Step 4 and the observed railway centerline in Step 6 to obtain the optimally estimated railway rail centerline.

[0015] Preferably, in Step 3, the present invention first extracts pairs of lines with high-confidence railway semantics, i.e., RTLP seeds, from the generated set of three-dimensional line segments, and defines them as S = {L i , L j}. Based on the angle θ, overlap degree o, and projection distance d, group the reconstructed three-dimensional line segments, and include the three-dimensional lines that satisfy Equation (1) in the candidate set of RTLP seeds:

[0016] {D∣θ(L i , L j ) < t θ , o(L i , L j ) > t o , d(L i , L j ) ∈ I} (1) In Equation (1), θ(L i , L j ), o(L i , L j ), and d(L i , L j ) respectively represent the included angle, overlap degree, and projection distance of the three-dimensional line segments L i and L j ; t θ and t o are respectively the thresholds for the angle and overlap degree. Since RTLP is usually parallel and highly overlapping, these two parameters can be flexibly set according to the actual accuracy requirements; I represents the threshold range of the projection distance, and a rough estimate of the width of RTLP is required, and its estimated value is denoted as , which can be directly obtained through railway construction standards or directly measured from the point cloud. Take one-third of as the error tolerance, that is, , thereby reducing the dependence on the initial value.

[0017] Preferably, in Step 3, for the candidate line pairs D = (Li , L j ) perform scoring to screen the best - matching three - dimensional line segments to form line pairs; assume that for a certain line pair D in the candidate set m the center line is C m , if the distance from the center line C k is less than then the score of D m can be calculated according to the Gaussian weight function, as shown in formula (2):

[0018]

[0019] In formula (2), 1 < k < n, n represents the number of candidate line pairs, 0 and respectively represent the mean and standard deviation of the Gaussian distribution used in the Gaussian weight function. The standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is Subsequently, sort in descending order according to the score S, and select the line pair with the highest score to add to the railway tracking line pair seed set. If the line segment in a certain line segment pair is used by a line segment pair with a higher ranking, it is directly discarded.

[0020] Preferably, in step 3, enhance the reliability of the RTLP seeds through texture alignment of RTLP. First, crop the image block containing the two - dimensional line segments to obtain the image block containing the line pair, and project the line pair onto at most four images containing these two line segments. To reduce the blurring of texture features caused by scale and rotation, correct the local image to ensure that the center line of RTLP passes horizontally through the center of the image and the width is half of the image width. Then, extract texture features from the image block. Use the global average pooling layer of ResNet as the basic feature describing the image block. Let f ∈ R n be the feature of RTLP, and extract the feature set f i (i = 1, 2, …, m) from the image block, where m is the number of features. Finally, use DBSCAN to cluster the features based on the cosine distance, and retain the largest cluster as the RTLP seed.

[0021] Preferably, in step 4, based on the idea of autonomous driving technology, predict the growth direction of the RTLP seeds according to the simulated train. The simulated train can be uniquely determined by the vector x containing the center point (x, y, z) representing the spatial position of the train, the attitude angles (ρ, k, φ) representing the driving direction of the train, and the train width ω, as shown in formula (3):

[0022] x = |x, y, z, ρ, κ, φ, ω| T ∈ R 7 . (3)

[0023] Taking the RTLP seed as the starting point of the simulated train, lane - keeping technology is used to simulate the automatic driving of the train, and predictions are made according to the inertial forward direction of the train. During the prediction process, the state at the next moment (k - moment) is jointly determined by the time step Δt and the current attitude of the train, and the calculation is shown in formula (4):

[0024]

[0025] In formula (4), represents the value of each component of the prediction value vector of the simulated train at the k - moment.

[0026] Preferably, in step 5, to select the most relevant images, using the sparse connection points generated by SfM (Structure from Motion), determine which images the current RTLP should be projected onto: 1) Search for the k three - dimensional connection points closest to the current RTLP; 2) Extract the indices of the images to which these points belong; 3) Count the number of connection points contained in each image, and select the top k images containing at least one connection point for two - dimensional rail line detection.

[0027] Preferably, in step 5, the Hough transform framework is used to detect the rail line in the image block, and the specific steps are as follows:

[0028] 1) Perform Canny edge detection on the image block, convert the result to the Hough space, and construct the accumulator matrix H;

[0029] 2) Select the top k peaks in H, and screen two peaks that satisfy the condition of formula (5):

[0030]

[0031] In formula (5), H i and H j are the accumulator matrices of the straight lines L i and L j in the Hough space respectively, ρ i and ρ j represent the shortest distances from the straight lines L i and L j to the origin of the image coordinate system respectively, θ i and θ j represent the angles between the straight lines L i and L j and the x - axis respectively, and t dis2 is the distance threshold, which can be directly determined by projecting the three - dimensional rail line width into the image.

[0032] 3) Transform the two peaks selected in the previous step into the image space to obtain the edges of two rail lines, and calculate the midline of the two edge lines to obtain the fitted two-dimensional rail center line.

[0033] Preferably, in step 6, the three-dimensional railway center line observation value is reconstructed based on the parallel constraint. In the multi-view images, for each pair of three-dimensional rail lines on each side, n1 and n2 corresponding two-dimensional straight lines can be found. Through the combination of these straight lines, at most n1×(n1 - 1) / 2 and n2×(n2 - 1) / 2 three-dimensional line segments can be reconstructed. To ensure the accuracy of the three-dimensional reconstruction, instead of directly using the least squares method for fitting, two representative three-dimensional line segments are selected from them, and they need to meet the following conditions:

[0034] 1) Parallelism: The two line segments should be approximately parallel, and the included angle is less than the set threshold;

[0035] 2) Supportability: The line segments need to have sufficient two-dimensional detection results as support.

[0036] Mathematically, it can be expressed by formula (6):

[0037]

[0038] In formula (6), i # , j # are the indexes of the line segments, θ(L i , L j ) represents the included angle between the three-dimensional line segments L i and L j , t θ is the angle threshold, S(L) is the score of the line segment, n' represents the number of three-dimensional line segments, 0 and respectively represent the mean and standard deviation of the Gaussian distribution used in the Gaussian weight function . The standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is

[0039] Finally, calculate the observation value of the simulated train described by the pair of three-dimensional line segments at the next moment, that is, at the k moment Its 7 components are determined by the average values of the positions, postures, and projection distances of the two three-dimensional line segments.

[0040] Preferably, in step 7, a discrete Kalman filter is used to fuse the observation value and the predicted value to update the train state, and the process is as follows:

[0041] 1) Initialize the observation estimate value and the covariance error matrix P = I 7×7 ;

[0042] 2) Prediction (see Step 4) and covariance error matrix prediction, as shown in Equation (7):

[0043] P k = FP k-1 F T + Q (7)

[0044] where P k is the covariance matrix of the prior error at time k; Q is the covariance matrix of the predicted value noise, which can be determined according to empirical values; F is the state transition matrix, which is the result of applying the Jacobian matrix linear transformation to the nonlinear state transition process, and its calculation is shown in Equation (8):

[0045]

[0046] where represents the value of the m-th row and n-th column of the F matrix.

[0047] 3) Three-dimensional railway centerline reconstruction, constructing the observed values (see Steps 5 and 6), and at the same time performing termination condition checks: If the reconstruction process does not converge (unable to correctly reconstruct any three-dimensional line segment from multi-view images) or overlaps with other RTLP seeds (half of the nearest RTLP is within the t dis distance of the RTLP that has been reconstructed in the previous filtering process), terminate the Kalman filter;

[0048] 4) Update the Kalman gain according to Equation (9):

[0049] K k = P k (P k + R) -1 (9)

[0050] In Equation (9), K k is the Kalman gain at time k, and R is the covariance matrix of the observed value noise, which can be determined according to empirical values;

[0051] 5) Update the predicted value according to Equation (10):

[0052]

[0053] 6) Update the prior covariance error matrix according to Equation (11):

[0054] P′ k = (I - K k )P k (11)

[0055] where P′ k represents the updated covariance error matrix at time k.

[0056] 7) Set k = k + 1 and perform Kalman filtering for the next moment.

[0057] As a priority, the present invention further includes verifying and analyzing the correctness and integrity of railway line detection using precision and recall, and verifying and analyzing the accuracy of three-dimensional railway centerline reconstruction using the RMSE distance from a point to the optimal fitting plane.

[0058] The present invention also provides an automatic extraction device for railway rail centerlines based on images, including:

[0059] A processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an automatic extraction method for railway rail centerlines based on images as described in the above technical solution.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] The automatic extraction method and device for railway rail centerlines based on images proposed by the present invention are based on the Kalman filtering framework. By simulating a train to construct a rail track line pair (RTLP) to track railway lines, it is divided into two parts: RTLP initialization and RTLP tracking. First, based on the reconstructed three-dimensional line segments in the scene, the RTLP is initialized based on geometric feature constraints and texture alignment based on DBSCAN clustering of depth features to form RTLP seeds. Then, based on the RTLP seeds, the growth direction of the RTLP seeds is predicted according to the inertial direction of the train to construct a predicted value. The two-dimensional railway centerline is detected in the image block based on the Hough framework, and three-dimensional reconstruction is performed based on parallel constraints to construct an observed value. Finally, based on the discrete Kalman filter, the observed value and the predicted value are fused, the state is updated, and the next Kalman filtering is performed until the RTLP overlaps or the observed value cannot be stably reconstructed. The results show that the method proposed by the present invention can reconstruct the railway centerline more completely and with high precision, filling the gap in the method for detecting and reconstructing three-dimensional line segments based on images that is not specifically designed for railway lines. Description of the Drawings

[0062] Figure 1 Is the flowchart of the method of the present invention;

[0063] Figure 2 Is the flowchart for generating RTLP seeds;

[0064] Figure 3 Is the schematic diagram of the three-dimensional railway centerline reconstruction method;

[0065] Figure 4Flowchart of railway line tracking based on Kalman filter;

[0066] Figure 5 Schematic diagram of the dataset;

[0067] Figure 6 Schematic diagram of the method for manually annotating the 3D railway centerline;

[0068] Figure 7 Graph of experimental test results;

[0069] Figure 8 Graph of the method for evaluating the reconstruction accuracy of the 3D railway centerline. Specific implementation mode

[0070] To facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention will be further described in detail below in conjunction with the accompanying drawings and implementation examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0071] Please refer to Figure 1 the flowchart. The automatic extraction method of the railway rail centerline based on images provided by the present invention includes the following steps:

[0072] Step 1: Perform image matching on the high-resolution UAV images of the input railway scene, restore their positions and postures, and generate image connection points.

[0073] Step 2: Extract the two-dimensional straight line segments in the image and perform three-dimensional straight line segment reconstruction.

[0074] Step 3: Geometrically align the three-dimensional straight line segments reconstructed in Step 2 based on elements such as angle, overlap degree, and projection distance, and construct RTLP seeds based on DBSCAN clustering of depth features.

[0075] As a preference, the present invention first extracts pairs of straight lines with high-confidence railway semantics, that is, RTLP (rail track line pair) seeds, from the generated set of three-dimensional straight line segments, which can also be defined as D = {L i , L j}, to provide a robust initial value for the Kalman filter. Considering the obvious geometric features of the railway line, the reconstructed three-dimensional line segments are grouped based on the angle θ, overlap degree o, and projection distance d, and the three-dimensional straight lines that satisfy formula (1) are included in the candidate set of RTLP seeds:

[0076] {D∣θ(L i , L j ) < t θ , o(L i , L j ) > t o, d(L i , L j ) ∈ I} (1)

[0077] In Equation (1), θ(L i , L j ), o(L i , L j ) and d(L i , L j ) represent the included angle, overlap degree, and projection distance of the three-dimensional line segments L i and L j respectively; t θ and t o are the thresholds for the angle and overlap degree respectively. Since RTLPs are usually parallel and highly overlapping, these two parameters can be flexibly set according to the actual accuracy requirements; I represents the threshold range of the projection distance, and a rough estimate of the width of the RTLP is required. The estimated value is denoted as and can be directly obtained through railway construction standards or directly measured from the point cloud. Take one-third of as the error tolerance, that is, thereby reducing the dependence on the initial value.

[0078] Considering that for a line segment L i , there may be multiple line segments that match it to form a line pair. Therefore, the candidate line pairs D = (L i , L j ) in the candidate set are scored according to the degree of geometric alignment, and the best-matched three-dimensional line segments are selected to form line pairs; let the center line of a certain line pair D m in the candidate set be C m . If the distance from it to the center line C k of other line pairs is less than , then the score of D m can be calculated according to the Gaussian weight function, as shown in Equation (2):

[0079]

[0080] In Equation (2), 1 < k < n, where n represents the number of candidate line pairs, 0 and represent the mean and standard deviation of the Gaussian distribution used in the Gaussian weight function respectively. The standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is, Subsequently, they are sorted in descending order according to the score S, and the line pair with the highest score is selected to be added to the railway tracking line pair seed set. If a line segment in a line pair is used by a line pair with a higher ranking, it is directly discarded.

[0081] Preferably, the reliability of the RTLP seeds is enhanced through the texture alignment of the RTLP. The process is shown in the appendixFigure 2 . First, crop the image block containing the two-dimensional line segment to obtain the image block containing the line pair, and project the line pair onto at most four images containing these two line segments. To reduce the blurring of texture features caused by scale and rotation, it is necessary to correct the local image to ensure that the center line of the RTLP passes horizontally through the center of the image and has a width of half of the image width. Then, extract the texture features from the image block. Use the global average pooling layer of ResNet as the basic feature to describe the image block. This network has been trained on a large number of public data sets and can effectively capture texture patterns, performing well even in the case of no labels. Therefore, no additional benchmark and training data are required. Let f ∈ R n be the feature of the RTLP, and extract the feature set f i (i = 1, 2, …, m) from the image block, where m is the number of features. Finally, cluster the features using DBSCAN based on the cosine distance, and retain the largest cluster as the RTLP seed.

[0082] Step 4: Predict the center line of the railway rail based on the inertial forward direction of the simulated train, and make the RTLP seed grow in both side directions.

[0083] Preferably, based on the idea of autonomous driving technology, the growth direction of the RTLP seed is predicted according to the simulated train. The simulated train can be uniquely determined by a vector x containing the center point (x, y, z) representing the spatial position of the train, the attitude angles (ρ, κ, φ) representing the driving direction of the train, and the train width ω, as shown in formula (3):

[0084] x = |x, y, z, ρ, κ, φ, ω| T ∈ R 7 . (3)

[0085] Take the RTLP seed as the starting point of the simulated train, use the lane keeping technology to simulate the autonomous driving of the train, and make predictions according to the inertial forward direction of the train. During the prediction process, the state at the next moment (k moment) is jointly determined by the time step Δt and the current attitude of the train, and the calculation is as shown in formula (4)

[0086]

[0087] In formula (4), is the value of each component of the prediction value vector of the simulated train at the k moment.

[0088] Step 5: Select the images used to generate the observation line segments of the railway rail center line based on the number of the same connection points, divide the images into blocks, and transform the edge features in each image block into the Hough space to extract the two-dimensional railway center line.

[0089] Preferably, to select the most relevant images, the sparse connection points generated by SfM (Structure from Motion) are used to determine which images the current RTLP should be projected onto: 1) Search for the k three-dimensional connection points closest to the current RTLP; 2) Extract the indices of the images to which these points belong; 3) Count the number of connection points contained in each image, and select the top k images containing at least one connection point for two-dimensional rail line detection.

[0090] In the image block, the Hough transform framework is used for rail line detection. The process is shown in the appendix Figure 3 . The specific steps are as follows:

[0091] 1) Perform Canny edge detection on the image block, convert the result to the Hough space, and construct the accumulator matrix H;

[0092] 2) Select the top k peaks in H and filter two peaks that satisfy the condition of formula (5):

[0093]

[0094] In formula (5), H i and H j are the accumulator matrices of the straight lines L i and L j in the Hough space respectively. ρ i and ρ j represent the shortest distances between the straight lines L i and L j and the origin of the image coordinate system respectively. θ i and θ j represent the angles between the straight lines L i and L j and the x-axis respectively. t dis2 is the distance threshold, which can be directly determined by projecting the three-dimensional rail line width into the image.

[0095] 3) Transform the two peaks selected in the previous step into the image space, obtain the edges of the two rail lines, and calculate the midline of the two edge lines to obtain the fitted two-dimensional rail center line.

[0096] Step 6: Perform three-dimensional reconstruction on the two-dimensional rail center line in step 5 under the parallel condition constraint.

[0097] Preferably, the reconstruction of the three-dimensional railway centerline observation values is based on parallel constraints. In the multi-view images, for each pair of three-dimensional rail lines on each side, n1 and n2 corresponding two-dimensional straight lines can be found. Through the combination of these straight lines, at most n1×(n1 - 1) / 2 and n2×(n2 - 1) / 2 three-dimensional line segments can be reconstructed. To ensure the accuracy of the three-dimensional reconstruction, instead of directly using the least squares method for fitting, two representative three-dimensional line segments are selected from them, and they need to meet the following conditions:

[0098] 1) Parallelism: The two line segments should be approximately parallel, and the included angle is less than the set threshold;

[0099] 2) Supportability: The line segments need to have sufficient two-dimensional detection results as support.

[0100] Mathematically, it can be expressed by formula (6):

[0101]

[0102] In formula (6), i # , j # is the index of the line segment, θ(L i , L j ) represents the included angle between the three-dimensional line segments L i and L j , t θ is the angle threshold, S(L) is the score of the line segment, n' represents the number of three-dimensional line segments, 0 and respectively represent the mean and standard deviation of the Gaussian distribution used in the Gaussian weight function . The standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is

[0103] Finally, calculate the observation value of the simulated train at the next moment (k moment) described by this pair of three-dimensional straight lines Its seven components can be determined by the average values of the positions, postures, and projection distances of the two three-dimensional straight lines.

[0104] Step 7: Based on the Kalman filter, fuse the predicted railway centerline in Step 4 and the observed railway centerline in Step 6 to obtain the optimally estimated railway centerline.

[0105] Preferably, a discrete Kalman filter is used to fuse the observed values and predicted values to update the train state. The process is shown in the appendix Figure 4 shown. Considering that the observation results may contain incorrect two-dimensional line segment detections or large three-dimensional reconstruction errors, the prediction process is prone to failure in some cases, especially when the train passes through a curve. Therefore, based on the Kalman filter, the observed values and predicted values are fused to obtain a high-precision railway centerline. The process is as follows:

[0106] 1) Initialize the observed estimate value And the covariance error matrix P = I 7×7 , I 7×7 represents the 7×7 identity matrix;

[0107] 2) Prediction (see step 4) and covariance error matrix prediction, as shown in formula (7):

[0108] P k = FP k-1 F T + Q (7)

[0109] where, P k is the covariance matrix of the prior error at time k; Q is the covariance matrix of the prediction value noise, which can be determined according to empirical values; F is the state transition matrix, which is the result of applying the Jacobian matrix linear transformation to the nonlinear state transition process, and its calculation is shown in formula (8):

[0110]

[0111] where, represents the value of the m-th row and n-th column of the F matrix.

[0112] 3) Three-dimensional railway centerline reconstruction, construct the observed value (see steps 5 and 6), and at the same time perform the termination condition check: If the reconstruction process does not converge (any three-dimensional line segment cannot be correctly reconstructed from the multi-view images) or overlaps with other RTLP seeds (half of the nearest RTLP is within the t dis distance of the RTLP that has been reconstructed in the previous filtering process), terminate the Kalman filter;

[0113] 4) Update the Kalman gain according to formula (9):

[0114] K k = P k (P k + R) -1 (9)

[0115] In formula (9), K k is the Kalman gain at time k, and R is the covariance matrix of the observed value noise, which can be determined according to empirical values;

[0116] 5) Update the predicted value according to formula (10):

[0117]

[0118] 6) Update the prior covariance error matrix according to formula (11):

[0119] P' k =(I - K k )P k (11)

[0120] wherein, P' k represents the covariance error matrix updated at time k.

[0121] 7) Set k = k + 1 and perform the Kalman filter for the next time step.

[0122] Preferably, in each step of prediction, there is no need to check the termination condition sequentially, especially when a small Δt is selected to ensure robustness. Instead, the termination condition can be checked uniformly when the iteration reaches a certain number of times. Once overlap or reconstruction degradation occurs, the system will return to the state checked last time to ensure the accuracy and robustness of the tracking process.

[0123] Step 8: The present invention uses 5 railway line datasets to test the performance of the method. The datasets are shown in the appendix Figure 5 . The correctness and integrity of the railway line detection of the present invention are verified and analyzed using precision, recall, and F - score. Manually annotate the true value of the 3D railway centerline. The annotation method is shown in the appendix Figure 6 , and the geometric accuracy of the railway line reconstruction of the present invention is verified and analyzed using the RMSE distance from a point to the optimal plane fitted by RANSAC. The index results are shown in Table 1.

[0124] Table 1 Index Analysis Table

[0125]

[0126] Appendix Figure 7 shows the railway line detection results under 5 datasets. Figure 7 As shown in Table 1, the method designed by the present invention does not extract incorrect railway lines in all 5 datasets. In terms of the integrity (recall rate) of railway line detection, the effect in dataset #4 is slightly worse. The reason is that there are many construction sections and the clarity of the rail lines in the images is poor, which limits the high - precision extraction of railway lines in the images. The recall rates of other datasets all reach above 0.95, and the results are good. Appendix Figure 8 shows the evaluation method and examples of the geometric accuracy of railway line reconstruction. The results are shown in Table 1. The method designed by the present invention has an RMSE within 70 mm in the horizontal direction and within 45 mm in the vertical direction for the railway centerline, showing better performance, which proves that it can achieve high - precision 3D railway centerline reconstruction.

[0127] On the other hand, the embodiment of the present invention also provides an automatic extraction device for railway rail centerlines based on images, including:

[0128] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for automatically extracting the center line of railway rails based on images as described in the above technical solution.

[0129] It should be understood that the parts not elaborated in detail in this specification all belong to the prior art.

[0130] It should be understood that the above description of the preferred embodiment is relatively detailed, and it should not be considered as a limitation to the protection scope of the invention patent. Under the inspiration of the present invention, those of ordinary skill in the art can also make substitutions or deformations without departing from the protection scope defined by the claims of the present invention, and all fall within the protection scope of the present invention. The scope of protection claimed by the present invention shall be subject to the appended claims.

Claims

1. A method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image, characterized in that: The following steps are involved: Step 1: Perform image matching on the high-resolution UAV images of the input railway scene, restore the position and posture of the multi-view images, and generate image connection points; Step 2, extracting the two-dimensional straight line segments in the image and reconstructing the initial three-dimensional straight line segments; Step 3, geometrically align the three-dimensional straight line segments in step 2 based on angle, overlap, and projection distance, and construct railway tracking line pair seeds based on DBSCAN clustering of deep features; Step 4, simulating the inertial forward direction of the train to predict the center line of the railway rail, so that the railway tracking line pair seeds grow in both directions; Step 5: during the growth process, an image for generating an observation line segment of the railway rail centerline is selected based on the number of adjacent three-dimensional connection points; after projecting the growth endpoints of the three-dimensional line segment onto the image, the edge features of each local image are converted into Hough space to extract the two-dimensional railway centerline; Step 6, reconstructing the center line of the multi-view two-dimensional rail railway in step 5 in three dimensions under the constraint of parallel conditions; Step 7: Based on Kalman filtering, the railway centerline predicted in step 4 and the railway centerline observed in step 6 are integrated to obtain the optimal estimated railway rail centerline.

2. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 3, the reconstructed initial 3D line segments are grouped based on the angle θ, overlap o and projection distance d, and the 3D straight lines satisfying formula (1) are included in the candidate set of railway tracing line pair seeds: {D∣θ(L i ,L j )<t θ ,o(L i ,L j )>t o ,d(L i ,L j )∈I} (1) In formula (1), θ(L i ,L j ), o(L i ,L j ) and d(L i ,L j ) represent the three-dimensional line segments L i and L j Angle, overlap and projection distance; t θ and t o are the thresholds of angle and overlap respectively, I represents the threshold range of projection distance, represents the estimated width of the railway trace pair seed; The candidate line pairs in the candidate set are sorted by the degree of geometric alignment D = (L i ,L j ) to score and select the best matching three-dimensional line segments to form line pairs; suppose a line pair D in the candidate set m The center line is C m , if the center line C of other line pairs k The distance is less than Then D m The score of can be calculated according to the Gaussian weight function, as shown in formula (2): In formula (2), 1 < k < n, where n represents the number of candidate line pairs, 0 and respectively represent the mean and standard deviation of the Gaussian distribution used in the Gaussian weight function. The standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is Subsequently, sort in descending order according to the score S, and select the line pair with the highest score to be added to the railway tracking line pair seed set. If the line segment in a certain line segment pair is used by a line segment pair with a higher ranking, it is directly discarded.

3. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 3, the railway tracing line pair seeds are texture aligned, which specifically includes: first, cropping the image block containing the two-dimensional line segment to obtain the image block containing the line pair, and projecting the line pair onto at most four images containing the two line segments; then correcting the local image to ensure that the center line of the railway tracing line pair seed passes horizontally through the center of the image and its width is half of the image width; then extracting texture features from the image block, using the global average pooling layer of ResNet as the basic feature to describe the image block, let f∈R n Extract the feature set f from the image block to obtain the features of the railway tracking line pair seed. i (i=1,2,…,m), where m is the number of features; finally, DBSCAN is used to cluster the features based on cosine distance, and the largest cluster is retained as the railway tracking line pair seed.

4. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 4, the simulated train is uniquely determined by the vector X containing the center point (X, y, z), attitude angle (ρ, κ, φ) and width ω, as shown in formula (3): x=|x,y,z,ρ,κ,φ,ω| T ∈R 7 . (3) The railway tracking line pair seed is used as the starting point of the simulated train, and the lane keeping technology is used to simulate the automatic driving of the train. The train's inertial forward direction is predicted. During the prediction process, the next state It is determined by the time step Δt and the current posture of the train, and is calculated as shown in formula (4): In formula (4), * k is the value of each component of the vector prediction value of the simulated train at time k.

5. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 5, the sparse connection points generated by SfM are used to determine which images the current railway tracing line pair seed should be projected onto, so as to select the most relevant image, which includes: 1) searching for the k three-dimensional connection points closest to the current railway tracing line pair seed; 2) extracting the index of the image to which these points belong; 3) counting the number of connection points contained in each image, and selecting the first k images containing at least one connection point for two-dimensional rail line detection.

6. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 5, the Hough transform framework is used to detect rail lines in the image block. The specific steps are as follows: 1) Perform Canny edge detection on the image block, convert the result into Hough space, and construct the accumulator matrix H; 2) Select the first k peaks in H and select the two peaks i that meet the conditions of formula (5) * ,j * : In formula (5), H i and H j The straight line L i and L j The accumulator matrix in Hough space, ρ i and ρ j Respectively represent the straight line L i and L j The shortest distance from the origin of the image coordinate system, θ i and θ j Respectively represent the straight line L i and L j Angle with the x-axis, t θ is the angle threshold, t dis2 is the distance threshold, which is directly determined by the projection of the 3D rail line width into the image; 3) The two peaks selected in the previous step are transformed into the image space to obtain the two rail line edges, and the midlines of the two edge lines are calculated to obtain the fitted two-dimensional rail centerline.

7. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 6, the 3D railway centerline observation values ​​are reconstructed based on the parallel constraint. In the multi-view image, for each 3D rail pair on each side, n1 and n2 corresponding 2D straight lines are found. Through the combination of these straight lines, at most n1×(n1-1) / 2 and n2×(n2-1) / 2 3D line segments are reconstructed. Two representative 3D line segments are selected from them, which meet the following conditions: 1) Parallelism: The angle between two line segments is less than the set threshold; 2) Supportability: The line segment has sufficient two-dimensional detection results as support, which can be expressed mathematically as formula (6): In formula (6), i # ,j # is the index of the three-dimensional line segment, θ(L i ,L j ) represents a three-dimensional line segment L i and L j The angle, t θ is the angle threshold, S(L) is the score of the line segment, n' represents the number of three-dimensional line segments, 0 and Represent the Gaussian weight function The mean and standard deviation of the Gaussian distribution used in , the standard deviation is set based on the 3σ criterion in the Gaussian distribution, that is Finally, the observation value of the simulated train described by the three-dimensional line segment at the next moment, i.e., moment k, is calculated. Its seven components are determined by the average values ​​of the position, attitude, and projection distance of two three-dimensional line segments.

8. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: In step 7, a discrete Kalman filter is used to fuse the observed value and the predicted value to obtain a high-precision railway centerline. The process is as follows: 1) Initialize the observation estimate and the covariance error matrix P = I 7×7 , I 7×7 represents the identity matrix with 7 rows and 7 columns; 2) The prediction and covariance error matrix prediction are shown in formula (7): P k =FP k-1 F T +Q (7) Among them, P k is the covariance matrix of the prior error at time k; Q is the covariance matrix of the predicted value noise, which is determined based on empirical values; F is the state transfer matrix, which is the result of applying the Jacobian matrix linear transformation to the nonlinear state transfer process. Its calculation is shown in formula (8): in, Represents the value of the mth row and nth column of the F matrix; 3) 3D railway centerline reconstruction, i.e. constructing observation values At the same time, the termination condition is checked: if the reconstruction process does not converge, that is, any 3D line segment cannot be correctly reconstructed from the multi-view image or overlaps with other railway tracing line pair seeds, that is, half of the nearest railway tracing line pair seeds have been reconstructed in the previous filtering process. dis Within the distance, terminate the Kalman filter; 4) Update the Kalman gain according to formula (9): K k =P k (P k +R) -1 (9) In formula (9), K k is the Kalman gain at time k, and R is the covariance matrix of the observation noise; 5) Update the predicted value according to formula (10): 6) Update the prior covariance error matrix according to formula (11): P′ k =(I-K k )P k (11) Among them, P′ k represents the covariance error matrix after updating at time k; 7) k=k+1, perform Kalman filtering at the next moment.

9. The method for automatic extraction and reconstruction of the centerline of a three-dimensional railway rail based on an image according to claim 1, characterized in that: It also includes the verification and analysis of the correctness and completeness of railway line detection using precision and recall, and the verification and analysis of the accuracy of three-dimensional railway centerline reconstruction using the RMSE distance from the point to the optimal fitting plane.

10. An image-based 3D railway rail centerline automatic extraction and reconstruction device, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an image-based three-dimensional railway rail centerline automatic extraction and reconstruction method as described in any one of claims 1 to 9.

Images