Running train mileage positioning method and device based on scene reconstruction

Abstract

The invention discloses a running train mileage positioning method and device based on scene reconstruction. The method comprises the following steps: detecting a semantic target with obvious features from a train running video shot by a monocular camera; target tracking is carried out on semantic targets in the train operation video containing the semantic target detection frame, and the semantic targets in each frame of image are marked with numbers; according to a train operation video containing a semantic target detection frame and a serial number and the optimized monocular SLAM model, camera poses and motion trails obtained after camera pose screening is carried out on prior information of train operation are obtained, and a three-dimensional point cloud map obtained after semantic segmentation is obtained through reconstruction; mileage calibration is carried out according to the camera pose, the movement track and the three-dimensional point cloud map, and running train mileage positioning information and a three-dimensional point cloud semantic map are obtained. According to the method, the train running mileage can be efficiently and accurately positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of railway infrastructure detection, and in particular to a method and device for locating the mileage of a running train based on scene reconstruction. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] With the rapid development of the railway industry, the importance of rail transportation has gradually increased, promoting inter-city connectivity and boosting regional economic development. As train speeds continue to increase, safety has become a constant concern for high-speed rail. To ensure operational safety, improve routine inspections and defect detection of railway infrastructure, and guide rail maintenance and repair efficiency, trains require a "time-space calibration system." This system provides unified time and space coordinates for all systems on the train, acting as the "headquarters" for all systems.

[0004] Before the concept of a "time-space calibration system" was proposed, mileage positioning systems typically performed this task. Traditional mileage positioning methods include wheel odometers, GPS positioning, and space-time positioning technology based on radio frequency (RF) tags. However, all of these methods suffer from various drawbacks: wheel odometers cannot handle mileage jumps; GPS positioning suffers from poor signal quality in mountainous areas or tunnels; RF tags are prone to aging, and deploying them along a full railway line requires a very large and expensive number of tags. Consequently, the railway industry urgently sought a method that was highly operational, cost-effective, and suitable for large-scale deployment, and thus turned its attention to the field of computer vision.

[0005] In the field of computer vision, odometry positioning technology primarily relies on offline 3D reconstruction algorithms or simultaneous localization and mapping (SLAM) technology. However, while both sparse reconstruction algorithms such as ORB-SLAM and dense reconstruction algorithms such as TANDEM and Neural Radiance Field (NERF) offer excellent performance in common low-speed scenarios, they still face the following challenges in specialized scenarios such as railway environments:

[0006] 1) Feature matching is prone to errors: Railway operating scenes are relatively simple, with numerous repetitive features, such as rails, utility poles, and ballast. The feature points extracted from these objects in the image have very similar feature descriptors, which can easily lead to feature point matching errors. Furthermore, faster driving speeds can result in fewer feature matching pairs between adjacent frames, affecting feature matching accuracy.

[0007] 2) Camera pose and trajectory don't match reality: Because the camera is placed at the front or rear of the train, which constantly moves along the rails, the train's trajectory is essentially sandwiched between the two rails in terms of yaw. In terms of roll, the train barely rotates. In terms of pitch, the train may climb or descend slightly, but the gradient is relatively small. The camera trajectory estimated by traditional methods may not be within the constraints of the rails, resulting in significant errors.

[0008] 3) Scale Drift: Due to the scale uncertainty of monocular cameras, each train pose estimation will have a certain degree of error. Train routes can be tens or even hundreds of kilometers long. As the distance traveled increases, the train's pose will experience significant scale drift. Furthermore, because railway tracks lack loops, existing methods for suppressing scale drift are ineffective. When a moving object returns to a previously reached position, this is called a "loop." Currently, most SLAM technologies use loop detection to suppress scale drift. The loop detection module determines whether the camera has returned to the same position as before and uses relevant algorithms to correct the error. However, trains generally move in one direction and do not return within a short period of time, making the loop detection module ineffective.

[0009] Most existing visual odometry technologies rely on multi-sensor fusion to overcome these issues. However, current trains are only equipped with cameras at the front of the train. Trains equipped with multiple sensors exist only in the laboratory and are not yet operational on the railway. Summary of the Invention

[0010] An embodiment of the present invention provides a method for locating the mileage of a running train based on scene reconstruction, for efficiently and accurately locating the running mileage of a train. The method includes:

[0011] Detect semantic objects with feature prominence greater than a preset value from each frame of a train running video captured by a monocular camera, and obtain a train running video containing a semantic object detection frame.

[0012] Track the semantic targets in the train running video containing the semantic target detection frame, label the semantic targets detected in each frame with the target number, and obtain the train running video containing the semantic target detection frame and the target number;

[0013] Based on a train running video containing semantic target detection frames and target numbers, and a pre-established optimized monocular SLAM model, the camera pose and motion trajectory after camera pose screening based on prior information of the train running are obtained, and a three-dimensional point cloud map after semantic segmentation is reconstructed; the optimized monocular SLAM model is a model that matches semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train running environment characteristics;

[0014] Mileage calibration is performed based on the camera pose, motion trajectory, and the reconstructed semantically segmented three-dimensional point cloud map to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

[0015] The embodiment of the present invention further provides a running train mileage positioning device based on scene reconstruction, which is used to achieve efficient and accurate positioning of train running mileage. The device includes: a target detection module, a target tracking module, a monocular SLAM module and a point cloud statistical processing module; wherein:

[0016] The target detection module is used to detect semantic targets with feature visibility greater than a preset value from each frame of the train running video captured by the monocular camera, and obtain a train running video containing the semantic target detection frame;

[0017] The target tracking module is used to track the semantic targets in the train running video containing the semantic target detection frame, mark the semantic targets detected in each frame with the target number, and obtain the train running video containing the semantic target detection frame and the target number;

[0018] A monocular vision SLAM module is used to obtain the camera pose and motion trajectory after camera pose screening based on the prior information of the train operation, and reconstruct a three-dimensional point cloud map after semantic segmentation based on the train operation video containing semantic target detection frames and target numbers, and a pre-established optimized monocular SLAM model; the optimized monocular SLAM model is a model that matches semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train operation environment characteristics;

[0019] The point cloud post-processing module is used to perform mileage calibration based on the camera pose, motion trajectory, and the reconstructed three-dimensional point cloud map after semantic segmentation, to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

[0020] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for locating the mileage of a running train based on scene reconstruction is implemented.

[0021] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for locating the mileage of a running train based on scene reconstruction is implemented.

[0022] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for locating the mileage of a running train based on scene reconstruction.

[0023] The beneficial effects of the running train mileage positioning solution based on scene reconstruction provided by the embodiment of the present invention are:

[0024] First of all, considering the technical problems in the prior art, such as the single railway scene mode, the feature points extracted from the image have very similar feature descriptors, and the fast running speed, which easily lead to feature point matching errors, the embodiment of the present invention adopts an optimized monocular SLAM model to realize the optical flow information of the image and the characteristics of the railway train operating environment, and sets the optical flow constraint adaptive threshold to alleviate the problem of feature point matching errors caused by the single railway scene mode and the fast running speed.

[0025] Secondly, the present invention utilizes the prior information that the train always moves on the rails and introduces it into the process of camera pose screening to obtain the camera pose more accurately and quickly. The camera pose and trajectory are consistent with the facts, thereby improving the accuracy of the train's spatial positioning.

[0026] Again, taking into account the technical problem in the prior art that the train running space positioning error accumulates infinitely due to the lack of loops in the railway running track, the embodiment of the present invention introduces semantic information, that is, detects stable targets in the railway environment, that is, semantic targets with feature visibility greater than a preset value, and marks the semantic targets with numbers through target tracking for auxiliary mileage positioning, thereby reducing the accumulation of train running space positioning errors, alleviating the scale drift problem, and improving the accuracy of train running space positioning.

[0027] In summary, the embodiments of the present invention achieve efficient and accurate positioning of train mileage, and establish a three-dimensional map with both mileage trajectory and semantic information, which is of great significance to railway mileage positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 Schematic diagram of the flow of a method for mileage positioning of a running train based on scene reconstruction in an embodiment of the present invention;

[0030] Figure 2This is a flow chart of a semantic map positioning technology based on monocular vision scene reconstruction in an embodiment of the present invention;

[0031] Figure 3 This is a flow chart of the optimized monocular SLAM algorithm in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram showing the comparison before and after feature point matching optimization in an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of reprojection error in an embodiment of the present invention;

[0034] Figure 6 Schematic diagram of the structure of a running train mileage positioning device based on scene reconstruction in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0036] The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant laws and regulations.

[0037] Figure 1 FIG. 1 is a flow chart of a method for locating the mileage of a running train based on scene reconstruction in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0038] Step 101: Detecting a semantic target with a feature significance greater than a preset value from each frame of a train running video captured by a monocular camera, and obtaining a train running video containing a semantic target detection frame;

[0039] Step 102: Tracking the semantic target in the train running video containing the semantic target detection frame, marking the semantic target detected in each frame with a target number, and obtaining the train running video containing the semantic target detection frame and the target number;

[0040] Step 103: Based on the train running video containing the semantic target detection frame and the target number, and the pre-established optimized monocular SLAM model, the camera pose and motion trajectory after the camera pose is screened by the prior information of the train running are obtained, and a three-dimensional point cloud map after semantic segmentation is reconstructed; the optimized monocular SLAM model is a model that matches the semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train running environment characteristics;

[0041] Step 104: Perform mileage calibration based on the camera pose, motion trajectory, and the reconstructed semantically segmented three-dimensional point cloud map to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

[0042] First of all, considering the technical problems in the prior art, such as the single railway scene mode, the feature points extracted from the image have very similar feature descriptors, and the fast running speed, which easily lead to feature point matching errors, the embodiment of the present invention adopts an optimized monocular SLAM model to realize the optical flow information of the image and the characteristics of the railway train operating environment, and sets the optical flow constraint adaptive threshold to alleviate the problem of feature point matching errors caused by the single railway scene mode and the fast running speed.

[0043] Secondly, the present invention utilizes the prior information that the train always moves on the rails and introduces it into the process of camera pose screening to obtain the camera pose more accurately and quickly. The camera pose and trajectory are consistent with the facts, thereby improving the accuracy of the train's spatial positioning.

[0044] Again, taking into account the technical problem in the prior art that the train running space positioning error accumulates infinitely due to the lack of loops in the railway running track, the embodiment of the present invention introduces semantic information, that is, detects stable targets in the railway environment, that is, semantic targets with feature visibility greater than a preset value, and marks the semantic targets with numbers through target tracking for auxiliary mileage positioning, thereby reducing the accumulation of train running space positioning errors, alleviating the scale drift problem, and improving the accuracy of train running space positioning.

[0045] In summary, the embodiments of the present invention achieve efficient and accurate positioning of train mileage, establishing a three-dimensional map that combines mileage trajectory and semantic information, which is of great significance for railway mileage positioning. The following is a detailed introduction to the running train mileage positioning method based on scene reconstruction.

[0046] The embodiment of the present invention proposes a method for locating the mileage of a running train based on scene reconstruction. The method only uses an on-board monocular camera as input, and uses prior information for optimization and constraints, aiming to propose a practical technology that can be put into railway operations. Currently, high-precision positioning algorithms based on monocular cameras, such as the ORB-SLAM series, have excellent performance in general scenarios. However, for special scenarios such as railway environments, there are still problems such as the relatively simple railway operation scene mode and the fast running speed, which lead to errors in feature matching, inconsistent camera pose and trajectory with the facts, and scale drift caused by loss of loop detection. The embodiment of the present invention organically combines the target detection algorithm, the multi-target tracking algorithm and the monocular SLAM algorithm, and proposes a semantic map spatial positioning method for railway environment scene reconstruction. The method is introduced in detail below.

[0047] In the embodiment of the present invention, the monocular visual semantic map positioning technology can include the following four technical modules: target detection module, target tracking module, monocular SLAM module and point cloud statistical processing module. The technical principle and the relationship between the four modules are as follows: Figure 2 As shown, Figure 2 This is a flow chart of the semantic map positioning technology based on monocular visual scene reconstruction in an embodiment of the present invention.

[0048] In the above-mentioned step 101, this step 101 can be implemented by a target detection module. That is, the embodiment of the present invention introduces a target detection module to obtain semantic information of the scene, detect targets with distinct features, and assist in positioning by using the target's position, thereby alleviating scale drift and minimizing error accumulation after loss of loop closure. Based on the backbone network architecture of YOLOv5, the present invention independently collected and organized several photos of railway contact network poles (telephone poles), calibrated them using LabelImg as a dataset, trained its own network weight model, and completed the detection of telephone poles in railway operation videos captured by a monocular camera.

[0049] As can be seen from the above, in one embodiment, a semantic target having a feature conspicuity greater than a preset value is detected from each frame of a train running video captured by a monocular camera, and a train running video containing a semantic target detection frame is obtained, including:

[0050] Based on the YOLOv5 backbone network architecture, several images containing semantic targets are collected and organized;

[0051] The semantic targets are calibrated using LabelImg and used as a dataset to train a network weight model to detect semantic targets in train running videos captured by a monocular camera, and obtain train running videos containing semantic target detection frames.

[0052] In one embodiment of the present invention, the semantic target with a feature prominence greater than a preset value may be one or a combination of a rail or a utility pole. Of course, the semantic target may also be ballast.

[0053] Step 102 can be implemented by a target tracking module. In step 102, the target tracking module uses the deep learning-based target tracking algorithm OCSORT to predict the target's position in the next frame using virtual trajectories and optical flow. A memory unit is also used to correct for inactive targets in a few frames due to occlusion. In one embodiment, the target tracking module tracks semantic targets in train operation videos containing semantic target detection frames, including using a memory unit to correct for inactive targets in target frames due to occlusion. This target tracking module completes the tracking of the utility poles detected by the previous module, i.e., it labels the detected utility poles with serial numbers.

[0054] The above step 103 can be performed by a monocular vision SLAM module (corresponding to Figure 2 In the above step 103, in the monocular vision SLAM module, the embodiment of the present invention uses the framework of ORB-SLAM. Figure 3 This is a flow chart of the optimized monocular SLAM algorithm in an embodiment of the present invention. Figure 3 The detailed process of the monocular vision SLAM module is demonstrated, and the optimization of the original algorithm is demonstrated. Figure 3 The content in the light green box is the optimization method of the present invention. Figure 3 As shown in the figure, the prior information of the railway environment is introduced into the feature extraction, feature matching and singular value decomposition of the intrinsic matrix, which effectively solves the problems of traditional SLAM methods in railway scenes:

[0055] (1) First, considering that the railway scene mode is single, the feature points extracted from the image have very similar feature descriptors, which can easily lead to feature point matching errors. The present invention sets an adaptive threshold for optical flow constraints based on the optical flow information of the image and the particularity of the railway operating environment to prevent feature point matching errors, such as Figure 3 As shown in the first green box.

[0056] (2) Secondly, since there is no loop in the railway running track, the error accumulates infinitely. The present invention introduces semantic information, that is, stable targets in the detection environment, such as telephone poles, etc., to assist in mileage correction and minimize error accumulation. That is, the embodiment of the present invention combines semantic information extraction and simultaneous localization and mapping (SLAM) technology to achieve auxiliary positioning, and uses common stable targets in the railway environment (such as telephone poles, etc.) to complete auxiliary positioning, aiming to alleviate the scale drift problem and improve positioning accuracy, such as Figure 3 As shown in the third green box.

[0057] (3) At the same time, the 3D map points reconstructed from the feature points detected on the target in the image are marked. The 3D map points are re-projected back to the 2D image, and the distance between the re-projected points and the original feature points is calculated. Points with a large difference between the re-projected points and the original feature points are eliminated. Random sampling is performed and repeated multiple times to obtain points with excellent "3D reconstruction consistency", such as Figure 3 As shown in the fourth green box in the figure.

[0058] (4) In addition, the algorithm of the present invention will lead to an increase in the amount of calculation, and real-time performance cannot be guaranteed on devices with poor performance. In order to increase the running speed, the present invention uses the prior information that the train is always moving on the rails. From the yaw direction, the train's movement trajectory is basically sandwiched between the two rails; from the roll direction, the train has almost no rotation; from the pitch direction, the train may have slight climbing and descending, but the slope change is small. The present invention introduces this prior information of train operation into the process of camera pose screening (camera pose screening combined with prior information is a step of the optimized monocular SLAM model) to obtain the camera pose more accurately and quickly, such as Figure 3 As shown in the second green box.

[0059] The following is a detailed introduction to the feature point matching scheme based on the adaptive threshold of the optical flow constraint mentioned in "(1)" above.

[0060] Railway operation scenes are relatively simple and contain a large number of repeated features, such as rails, utility poles, and ballast. The ORB feature points extracted from these objects in the image have very similar feature descriptors, which can easily lead to feature point matching errors. Figure 4 Schematic diagram showing the comparison before and after feature point matching optimization in an embodiment of the present invention. Figure 4 As shown in (a) of the figure, there is a pair of clearly mismatched feature points. The selection and matching of feature points is crucial in monocular vision SLAM. Such mismatching can lead to chain reactions of errors in subsequent camera pose and triangulation map building. The consequences are extremely serious, and constraints and optimization are necessary.

[0061] Optical flow can be viewed as a pixel displacement vector field from the first frame to the second frame, which gives the direction and speed of motion of each pixel between two consecutive frames. For the two-dimensional velocity field formed by all pixels in the image, the motion vector of each pixel can be understood as an optical flow, and all optical flows constitute the optical flow field. Although there are widespread cases where the optical flow field and the motion field are inconsistent, in the running scenario of a continuously moving train, the embodiment of the present invention assumes that the difference between the optical flow field and the motion field is not very large, and the image optical flow field can almost express the motion information expressed by the motion field.

[0062] Before finding the optical flow field of an image, the following two assumptions must be mentioned:

[0063] 1) Constant brightness. That is, when the same object moves between different frames, its brightness itself does not change.

[0064] 2) Both time and motion are continuous. That is, the change of time will not cause a drastic change in the target position, and the target position changes little between adjacent frames.

[0065] Let the irradiance of point (x, y) in the image at time t be I(x, y, t). For a very short time interval Δt, the first assumption is:

[0066] I(x,y,t)=I(x+Δx,y+Δy,t+Δt) (1)

[0067] Based on the second assumption, using Taylor expansion on equation (1), we get:

[0068]

[0069] Here, ε represents a second-order infinitesimal term and can be ignored. Then divide Δt on both sides of the equation, set Δt→0, and replace Δt with dt to obtain:

[0070]

[0071] The above formula can be simplified as:

[0072] I x u+I y v+I t =0 (4)

[0073] Among them, I x , I y and I t Both can be obtained from image data, and (u,v) is the required optical flow vector.

[0074] In the scenario where the train is running forward, the optical flow field can be approximately understood as a motion field. At locations where the motion changes are obvious (such as the edge of the image), the distance between the feature points of two adjacent frames will be farther; conversely, at locations where the motion changes are not obvious (such as the center of the image), the feature points of two adjacent frames will be closer. Based on the optical flow value at the feature point, the present invention sets an adaptive matching threshold T(x, y). Only when the distance on the image between the feature point pair (the feature point pair can refer to the feature point A' that matches the feature point A in the previous frame image in the adjacent frame image) (the distance refers to the pixel coordinate distance between the two feature points A and A' in the unified image coordinate system, that is, the moving distance of the feature point pixels between adjacent frame images) is less than the threshold, will the pairing be successful.

[0075] As can be seen from the above, in one embodiment, based on the train running video containing semantic target detection boxes and target numbers, and the pre-established optimized monocular SLAM model, reconstructing a semantically segmented 3D point cloud map may include:

[0076] Obtain the optical flow information of each semantic target feature point in each frame of the train running video;

[0077] According to the optical flow information at each semantic target feature point and the pre-established adaptive threshold model of the optical flow constraint, the adaptive threshold of the optical flow constraint at each semantic target feature point is obtained;

[0078] According to the optical flow constraint adaptive threshold at each semantic target feature point, the semantic target feature points are screened to obtain feature point pairs between adjacent frame images that are smaller than the optical flow constraint adaptive threshold;

[0079] Based on the feature point pairs between adjacent frame images that are less than the adaptive threshold of the optical flow constraint, the three-dimensional point cloud map after semantic segmentation is reconstructed.

[0080] The adaptive threshold of optical flow constraint can be expressed as:

[0081]

[0082] Where T(x,y) is the adaptive threshold of the optical flow constraint, λ1 and λ2 are coefficients set according to the actual working conditions, which can be tested and adjusted according to changes in the actual working conditions, and (Δu, Δv) represents the optical flow vector at the point (x,y).

[0083] As can be seen from the above, in one embodiment, obtaining the optical flow constraint adaptive threshold value at each semantic target feature point based on the optical flow information at each semantic target feature point and a pre-established adaptive threshold model of optical flow constraint may include: obtaining the optical flow constraint adaptive threshold value at each semantic target feature point according to the adaptive threshold model of optical flow constraint as shown in the above formula (5):

[0084]

[0085] Figure 4 (b) in the figure shows the matching results after the introduction of the adaptive threshold, which effectively avoids the feature point matching errors caused by the presence of a large number of repeated features in the railway operation scene and improves the accuracy of the algorithm.

[0086] The following is a detailed introduction to the scheme for three-dimensional reconstruction based on the feature points with excellent "three-dimensional reconstruction consistency" mentioned in "(3)" above.

[0087] The quality of feature points plays a crucial role in 3D reconstruction. This embodiment of the present invention proposes a metric for measuring feature point quality—"3D reconstruction consistency"—to select higher-quality feature points for 3D reconstruction and camera pose estimation. Specifically, in one embodiment, after reconstructing a semantically segmented 3D point cloud map, the following steps may also be performed:

[0088] The camera motion trajectory estimated by the 3D point cloud map frame by frame is reprojected onto the image plane;

[0089] The distance between the new projection point and the original feature point on the image plane is calculated as the reprojection error;

[0090] Eliminate feature points whose reprojection error is greater than a preset error threshold, and obtain feature points whose reprojection error is less than or equal to the preset error threshold;

[0091] The feature points whose reprojection error is less than or equal to the preset error threshold are reused to reconstruct the semantically segmented 3D point cloud map and camera pose estimation.

[0092] Specifically, Figure 5 is a schematic diagram of reprojection error in an embodiment of the present invention, as shown in FIG. Figure 5 As shown in Figure 1, the camera trajectory estimated from the 3D point cloud for that frame is reprojected onto the image plane. The distance between the newly projected points and the original feature points on the image plane (2D image) is calculated, which is called the reprojection error. Points with excessively large reprojection errors are removed, and the remaining feature points are reused for 3D mapping and camera pose estimation. This process can be iterated multiple times.

[0093] For a 3D map point P = (X, Y, Z) T And the feature point p2=(u,v) in the plane image T , and the reprojection point In homogeneous coordinates, we have the following relationship:

[0094]

[0095] Among them, K represents the camera internal parameter matrix, [R 3×3 t 3×1 ] represents the external parameter matrix of the camera, u, v represent the two-dimensional coordinate information of the feature point, X, Y, Z represent the three-dimensional coordinate information of the feature point in the three-dimensional point cloud map, and s represents the depth information of the reprojected pixel point (new projection point) in the image.

[0096] Specify a threshold T reproj , when the reprojection error vector

[0097]

[0098] When the 2-norm of is less than this threshold, that is:

[0099] ||e||2<T reproj (8)

[0100] It can be considered that this feature point is a feature point with excellent "three-dimensional reconstruction consistency".

[0101] From the above, it can be seen that in one embodiment, feature points whose reprojection errors are greater than a preset error threshold are eliminated to obtain feature points whose reprojection errors are less than or equal to the preset error threshold, including: when the 2-norm of the reprojection error vector is less than the preset error threshold, feature points whose reprojection errors are less than or equal to the preset error threshold are obtained; wherein the reprojection error vector is determined according to the reprojection error vector model shown in the above formula (7), wherein e is the reprojection error vector, K represents the camera intrinsic parameter matrix, [R 3×3 t 3×1 ] represents the external parameter matrix of the camera, s represents the image depth, and X, Y, and Z are the three-dimensional coordinate information of the feature points in the three-dimensional point cloud map.

[0102] It is worth mentioning that the above process is essentially different from the minimization of reprojection error to solve the PnP problem, which is solved by adjusting the camera extrinsic parameters [R 3×3 t 3×1 ], so that the reprojection error of all feature points is minimized, that is:

[0103]

[0104] It still assumes that every feature point is reliable, but errors in triangulation and the RANSAC method lead to errors in 3D map points and camera pose estimation. However, the mathematical principle of triangulation is simple and straightforward: as long as the input feature points are reliable, the 3D map points obtained by triangulation are also reliable. Therefore, the optimization of this invention aims to select feature points of higher quality, which is fundamentally different from solving the PnP problem by minimizing reprojection error.

[0105] Of course, multiple iterations may result in a reduction in the number of feature points used for mapping, and the time complexity of the algorithm will also increase. Therefore, it is recommended to randomly eliminate some points when eliminating. That is, in one embodiment, feature points whose reprojection errors are greater than a preset error threshold are eliminated to obtain feature points whose reprojection errors are less than or equal to the preset error threshold, including: eliminating a preset proportion of feature points among the feature points whose reprojection errors are greater than the preset error threshold to obtain feature points whose reprojection errors are less than or equal to the preset error threshold.

[0106] In short, the monocular SLAM module can obtain the camera's motion trajectory and a semantically segmented 3D point cloud map. The above step 104 can be performed by a point cloud post-processing module (corresponding to Figure 2The purpose of the present invention in designing this point cloud post-processing module is that there are always some inevitable errors in monocular vision SLAM, such as: the interval between two frames in the triangulation method is too short, resulting in a too short baseline distance, and the error will increase significantly. The monocular SLAM module uses basic statistical knowledge to complete the subsequent processing and visualization of the three-dimensional point cloud. The monocular vision semantic map technology proposed in the present invention combines the above four modules into a Figure 2 The modes in the system are connected in series to complete the mileage positioning of railway trains and the drawing of semantic maps along the route.

[0107] In summary, for railway scenarios, the present invention integrates semantic information into 3D reconstruction technology, proposing a monocular visual semantic map based on prior information. This technology optimizes the main issues associated with 3D reconstruction in railway environments, mitigating scale drift by locating stable semantic targets. It also mitigates the problem of feature point matching errors caused by the single mode and high speed of railway scenes by setting adaptive thresholds and filtering feature points based on "3D reconstruction consistency." A 3D map combining mileage trajectory and semantic information has been established, which is of great significance for railway mileage positioning.

[0108] The present invention also provides a device for locating the mileage of a running train based on scene reconstruction, as described in the following embodiments. Because the principles of this device are similar to those of the method for locating the mileage of a running train based on scene reconstruction, the implementation of this device can be referenced to the implementation of the method for locating the mileage of a running train based on scene reconstruction, and any repetitions will not be repeated.

[0109] Figure 6 FIG. 1 is a structural diagram of a running train mileage positioning device based on scene reconstruction in an embodiment of the present invention. Figure 6 As shown, the device includes: a target detection module 01, a target tracking module 02, a monocular SLAM module 03 and a point cloud statistics processing module 04; wherein:

[0110] The object detection module 01 is used to detect semantic objects with feature visibility greater than a preset value from each frame of the train running video captured by the monocular camera, and obtain the train running video containing the semantic object detection frame;

[0111] The target tracking module 02 is used to track the semantic target in the train running video containing the semantic target detection frame, mark the semantic target detected in each frame with a target number, and obtain the train running video containing the semantic target detection frame and the target number;

[0112] Monocular vision SLAM module 03 is used to obtain the camera pose and motion trajectory after camera pose screening based on the prior information of the train operation based on the train operation video containing the semantic target detection frame and target number, and the pre-established optimized monocular SLAM model, and reconstruct a three-dimensional point cloud map after semantic segmentation; the optimized monocular SLAM model is a model that matches semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train operation environment characteristics;

[0113] The point cloud post-processing module 04 is used to perform mileage calibration based on the camera pose and motion trajectory after pose screening, and the reconstructed three-dimensional point cloud map after semantic segmentation, to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

[0114] In one embodiment, the monocular vision SLAM module is specifically used to:

[0115] Obtain the optical flow information of each semantic target feature point in each frame of the train running video;

[0116] According to the optical flow information at each semantic target feature point and the pre-established adaptive threshold model of the optical flow constraint, the adaptive threshold of the optical flow constraint at each semantic target feature point is obtained;

[0117] According to the optical flow constraint adaptive threshold at each semantic target feature point, the semantic target feature points are screened to obtain feature point pairs between adjacent frame images that are smaller than the optical flow constraint adaptive threshold;

[0118] Based on the feature point pairs between adjacent frame images that are less than the adaptive threshold of the optical flow constraint, the three-dimensional point cloud map after semantic segmentation is reconstructed.

[0119] In one embodiment, according to the optical flow information at each semantic target feature point and a pre-established adaptive threshold model of optical flow constraint, obtaining the adaptive threshold of optical flow constraint at each semantic target feature point includes: obtaining the adaptive threshold of optical flow constraint at each semantic target feature point according to the following adaptive threshold model of optical flow constraint:

[0120]

[0121] Where T(x,y) is the adaptive threshold of the optical flow constraint, λ1 and λ2 are coefficients set according to the actual working conditions, which can be tested and adjusted according to changes in the actual working conditions, and (Δu, Δv) represents the optical flow vector at the point (x,y).

[0122] In one embodiment, the monocular vision SLAM module is further used to:

[0123] The camera motion trajectory estimated from the 3D point cloud map in this frame is re-projected onto the image plane;

[0124] The distance between the new projection point and the original feature point on the image plane is calculated as the reprojection error;

[0125] Eliminate feature points whose reprojection error is greater than a preset error threshold, and obtain feature points whose reprojection error is less than or equal to the preset error threshold;

[0126] The feature points whose reprojection error is less than or equal to the preset error threshold are reused to reconstruct the semantically segmented 3D point cloud map and camera pose estimation.

[0127] In one embodiment, removing feature points whose reprojection errors are greater than a preset error threshold to obtain feature points whose reprojection errors are less than or equal to the preset error threshold includes: obtaining feature points whose reprojection errors are less than or equal to the preset error threshold when the 2-norm of the reprojection error vector is less than the preset error threshold; wherein the reprojection error vector is determined according to the following reprojection error vector model:

[0128]

[0129] Among them, e is the reprojection error vector, K represents the camera intrinsic parameter matrix, [R 3×3 t 3×1 ] represents the extrinsic parameter matrix of the camera, s represents the depth information of the reprojected pixel in the image, u, v represent the two-dimensional coordinate information of the feature point, and X, Y, Z are the three-dimensional coordinate information of the feature point in the three-dimensional point cloud map.

[0130] In one embodiment, feature points whose reprojection errors are greater than a preset error threshold are eliminated to obtain feature points whose reprojection errors are less than or equal to the preset error threshold, including: eliminating a preset proportion of feature points from the feature points whose reprojection errors are greater than the preset error threshold to obtain feature points whose reprojection errors are less than or equal to the preset error threshold.

[0131] In one embodiment, the target detection module is specifically configured to:

[0132] Based on the YOLOv5 backbone network architecture, several images containing semantic targets are collected and organized;

[0133] The semantic targets are calibrated using LabelImg and used as a dataset to train a network weight model to detect semantic targets in train running videos captured by a monocular camera, and obtain train running videos containing semantic target detection frames.

[0134] In one embodiment, the target tracking module is specifically configured to use a memory unit to correct a situation in which the target is inactive in the target frame due to being occluded.

[0135] In one embodiment, the semantic target whose feature prominence is greater than a preset value is one of a rail and a utility pole, or a combination thereof.

[0136] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for locating the mileage of a running train based on scene reconstruction is implemented.

[0137] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for locating the mileage of a running train based on scene reconstruction is implemented.

[0138] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for locating the mileage of a running train based on scene reconstruction.

[0139] The beneficial effects of the running train mileage positioning solution based on scene reconstruction provided by the embodiment of the present invention are:

[0140] First of all, considering the technical problems in the prior art, such as the single railway scene mode, the feature points extracted from the image have very similar feature descriptors, and the fast running speed, which easily lead to feature point matching errors, the embodiment of the present invention adopts an optimized monocular SLAM model to realize the optical flow information of the image and the characteristics of the railway train operating environment, and sets the optical flow constraint adaptive threshold to alleviate the problem of feature point matching errors caused by the single railway scene mode and the fast running speed.

[0141] Secondly, the present invention utilizes the prior information that the train always moves on the rails and introduces it into the process of camera pose screening to obtain the camera pose more accurately and quickly. The camera pose and trajectory are consistent with the facts, thereby improving the accuracy of the train's spatial positioning.

[0142] Again, taking into account the technical problem in the prior art that the train running space positioning error accumulates infinitely due to the lack of loops in the railway running track, the embodiment of the present invention introduces semantic information, that is, detects stable targets in the railway environment, that is, semantic targets with feature visibility greater than a preset value, and marks the semantic targets with numbers through target tracking for auxiliary mileage positioning, thereby reducing the accumulation of train running space positioning errors, alleviating the scale drift problem, and improving the accuracy of train running space positioning.

[0143] In summary, the embodiments of the present invention achieve efficient and accurate positioning of train mileage (including spatial position), and establish a three-dimensional map with both mileage trajectory and semantic information, which is of great significance to railway mileage positioning.

[0144] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0148] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for locating the mileage of a running train based on scene reconstruction, characterized in that: include: Detecting semantic targets with feature prominence greater than a preset value from each frame of the train running video captured by the monocular camera, and obtaining a train running video containing a semantic target detection frame; Tracking the semantic target in the train running video containing the semantic target detection frame, marking the semantic target detected in each frame with a target number, and obtaining the train running video containing the semantic target detection frame and the target number; According to the train running video containing the semantic target detection frame and the target number, and the pre-established optimized monocular SLAM model, the camera pose and motion trajectory after the camera pose screening are obtained based on the prior information of the train running, and the three-dimensional point cloud map after semantic segmentation is reconstructed; the optimized monocular SLAM model is a model that matches the semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train running environment characteristics; According to the camera posture, motion trajectory, and the reconstructed three-dimensional point cloud map after semantic segmentation, mileage calibration is performed to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

2. The method according to claim 1, characterized in that Based on the train running video containing semantic target detection boxes and target numbers, and the pre-established optimized monocular SLAM model, the semantically segmented 3D point cloud map is reconstructed, including: Obtain the optical flow information of each semantic target feature point in each frame of the train running video; According to the optical flow information at each semantic target feature point and the pre-established adaptive threshold model of the optical flow constraint, the adaptive threshold of the optical flow constraint at each semantic target feature point is obtained; According to the optical flow constraint adaptive threshold at each semantic target feature point, the semantic target feature points are screened to obtain feature point pairs between adjacent frame images that are smaller than the optical flow constraint adaptive threshold; Based on the feature point pairs between adjacent frame images that are less than the adaptive threshold of the optical flow constraint, the three-dimensional point cloud map after semantic segmentation is reconstructed.

3. The method according to claim 2, characterized in that According to the optical flow information at each semantic target feature point and the pre-established adaptive threshold model of the optical flow constraint, the adaptive threshold of the optical flow constraint at each semantic target feature point is obtained, including: obtaining the adaptive threshold of the optical flow constraint at each semantic target feature point according to the following adaptive threshold model of the optical flow constraint: Among them, T(x, y) is the adaptive threshold of the optical flow constraint, λ1 and λ2 are coefficients set according to the actual working conditions, and (Δu, Δv) represents the optical flow vector at the point (x, y).

4. The method according to claim 1, characterized in that After reconstructing the 3D point cloud map after semantic segmentation, it also includes: The camera motion trajectory estimated by frame from the 3D point cloud map is reprojected onto the image plane; The distance between the new projection point and the original feature point on the image plane is calculated as the reprojection error; Eliminate feature points whose reprojection errors are greater than a preset error threshold, and obtain feature points whose reprojection errors are less than or equal to the preset error threshold; The feature points whose reprojection error is less than or equal to the preset error threshold are reused to reconstruct the 3D point cloud map and camera pose estimation after semantic segmentation.

5. The method according to claim 4, characterized in that Eliminating feature points whose reprojection errors are greater than a preset error threshold, and obtaining feature points whose reprojection errors are less than or equal to the preset error threshold, comprising: when the 2-norm of the reprojection error vector is less than the preset error threshold, obtaining feature points whose reprojection errors are less than or equal to the preset error threshold; wherein the reprojection error vector is determined according to the following reprojection error vector model: Among them, e is the reprojection error vector, K represents the camera intrinsic parameter matrix, [R 3×3 t 3×1 ] represents the external parameter matrix of the camera, s represents the depth information of the reprojected pixel in the image, u, v represent the two-dimensional coordinate information of the feature point, and X, Y, Z are the three-dimensional coordinate information of the feature point in the three-dimensional point cloud map.

6. The method according to claim 5, characterized in that Eliminating feature points whose reprojection errors are greater than a preset error threshold to obtain feature points whose reprojection errors are less than or equal to the preset error threshold, including: eliminating a preset proportion of feature points from feature points whose reprojection errors are greater than the preset error threshold to obtain feature points whose reprojection errors are less than or equal to the preset error threshold.

7. The method according to claim 1, characterized in that Detecting semantic targets whose feature prominence is greater than a preset value from each frame of the train running video captured by a monocular camera, and obtaining a train running video containing a semantic target detection frame, including: Based on the YOLOv5 backbone network architecture, several images containing semantic targets are collected and sorted; The semantic targets are calibrated using LabelImg and used as a data set to train a network weight model to complete the detection of semantic targets in the train running video taken by a monocular camera, and obtain the train running video containing the semantic target detection box.

8. The method according to claim 1, characterized in that Tracking semantic targets in train running videos containing semantic target detection frames includes using memory units to correct the situation in which the target is inactive in the target frame due to being occluded.

9. The method according to claim 1, characterized in that The semantic target whose feature prominence is greater than a preset value is one or a combination of rails and electric poles.

10. A running train mileage positioning device based on scene reconstruction, characterized in that: include: Target detection module, target tracking module, monocular SLAM module and point cloud statistical processing module; among which: The target detection module is used to detect a semantic target with a feature prominence greater than a preset value from each frame of the train running video captured by the monocular camera, and obtain a train running video containing a semantic target detection frame; The target tracking module is used to track the semantic target in the train running video containing the semantic target detection frame, mark the semantic target detected in each frame image with the target number, and obtain the train running video containing the semantic target detection frame and the target number; The monocular vision SLAM module is used to obtain the camera pose and motion trajectory after the camera pose is screened based on the prior information of the train operation according to the train operation video containing the semantic target detection frame and the target number, and the pre-established optimized monocular SLAM model, and reconstruct the three-dimensional point cloud map after semantic segmentation; the optimized monocular SLAM model is a model that matches the semantic target feature points in adjacent frame images through the optical flow information of the image and the optical flow constraint adaptive threshold set by the train operation environment characteristics; The point cloud post-processing module is used to perform mileage calibration according to the camera posture, motion trajectory, and the reconstructed three-dimensional point cloud map after semantic segmentation, so as to obtain the mileage positioning information of the running train and the three-dimensional point cloud semantic map containing the mileage positioning information.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images