A 3D track line detection method based on long-short camera fusion

Abstract

The application discloses a kind of 3D track line detection methods based on long-short focus camera fusion, belong to track line detection technical field.The method of the application includes: obtaining long-short focus image pair of synchronous acquisition, depth image is generated by stereo matching;Respectively extract the features of long-short focus image and depth image, and with depth feature as query, with long-short focus image feature as key and value, fusion is carried out by cross attention mechanism, and depth fusion feature is obtained;Based on predefined 3D space anchor point sequence, it is projected to depth fusion feature and feature is extracted, the spatial offset of each anchor point is predicted by regression network, and then the continuous three-dimensional track line coordinate sequence is decoded and output.The application effectively solves the limitation of monocular vision in depth perception, realizes the high-precision, robust three-dimensional reconstruction of track line in complex environment, and can be directly used for environmental perception of train automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of track line detection technology, and specifically relates to a 3D track line detection method based on the fusion of long and short focal length cameras. Background Technology

[0002] In the field of rail transit environmental perception, existing methods have significant limitations in their ability to perceive the three-dimensional structure of track under complex operating conditions. Especially during actual operation, factors such as changes in lighting, weather conditions, and background interference make it difficult for traditional monocular vision-based detection methods to achieve stable and accurate long-distance three-dimensional track structure reconstruction, severely restricting the safety of autonomous train operation.

[0003] Current image-based 3D detection methods mostly rely on independent, multi-stage processing flows, including image preprocessing, 2D feature extraction, and 3D coordinate regression modules. This results in complex system structures, low information transmission efficiency, and significant ambiguity in 3D coordinate prediction when there is a lack of prior knowledge of the track structure. Especially in complex scenarios such as track switch areas and multi-track intersection areas, existing methods struggle to accurately model the spatial topological relationships of the tracks, thus limiting the accuracy of 3D track line detection.

[0004] Current track detection technologies largely focus on recognition and segmentation at the two-dimensional image level, such as extracting track regions from images using semantic segmentation, curve fitting, or feature matching. However, with the increasing automation of rail transit, relying solely on two-dimensional image information is insufficient to meet the train's precise need for perceiving the three-dimensional structure of the road ahead. A detection solution capable of directly outputting three-dimensional track position information is urgently needed. For example, CN110516532B proposes a UAV railway track recognition method based on pulse-coupled neural networks and Bezier curve fitting, enabling the extraction and fitting of track segments from images; CN113221701A extracts track edge features within a preset distance in front of the vehicle and determines the start and end positions of the track segments by combining the travel direction; while CN102663352A performs Hough transform and distance calculation on track contour points to complete track recognition and ranging. These methods have achieved good results in two-dimensional track recognition, but none have overcome the limitations of two-dimensional image representation, failing to provide the actual position and geometric structure of the track in three-dimensional space.

[0005] Existing track inspection solutions suffer from significant shortcomings in 3D perception capabilities and system architecture when dealing with complex railway operating environments. Firstly, regarding 3D information acquisition, traditional monocular vision solutions can only provide 2D pixel-level track positioning, failing to directly acquire the track's 3D spatial coordinates. This makes it difficult to accurately reconstruct the actual spatial orientation of the track in turnout areas, curves, and long-distance scenarios. Secondly, at the feature representation level, existing methods lack effective modeling of the unique spatial continuity and geometric constraints of track lines. Under complex imaging conditions such as sudden changes in illumination and rain / snow interference, problems such as track line breaks and positional shifts easily occur, severely affecting the stability of detection. Furthermore, at the system design level, current technologies often employ a separate processing flow, with 2D feature extraction and 3D structure reconstruction being independent of each other. This leads to inconsistencies between the optimization objectives of feature representation and the final 3D output, hindering the overall system performance improvement and practical application value.

[0006] To address the aforementioned shortcomings of existing technologies, this invention proposes a 3D track line detection method based on the fusion of long and short focal length cameras. By constructing a multi-focal-length vision system, it achieves direct mapping from two-dimensional image features to three-dimensional track space, effectively solving the problem of the lack of three-dimensional spatial perception capability in traditional methods. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a 3D track line detection method based on the fusion of long and short focal length cameras. This method does not rely on multi-stage image enhancement or independent 3D reconstruction modules. Instead, it generates a high-precision disparity map and extracts depth features by constructing a stereo matching relationship between long and short focal length images, achieving end-to-end high-precision detection of 3D track structures from multi-focal length images. It solves the following problems in existing technologies: establishing a stable multi-source image feature fusion mechanism under complex lighting and multi-scale scenes; achieving deep fusion of 2D image features and 3D spatial positions based on disparity estimation; and maintaining robustness and geometric consistency in detection under complex scenes such as long distances and multiple tracks.

[0008] This invention provides a 3D track line detection method based on the fusion of long and short focal length cameras, the specific steps of which are as follows: Acquire image pairs, including telephoto and short-focus images, from a synchronously calibrated telephoto camera and a short-focus camera. Based on the image pairs, stereo matching and depth calculation are performed to obtain a depth image; Features of the telephoto image, short-focus image, and depth image are extracted, and the depth features are fused with the features of the telephoto and short-focus images using an attention mechanism to obtain strong representation features of depth perception information. Based on the strong representational features of the depth perception information, a trajectory coordinate prediction network for 3D trajectory line recognition is established. Construct a loss function for the orbital line coordinate prediction network, and train the orbital line coordinate prediction network based on the loss function; then construct a 3D orbital line detection model based on the trained orbital line coordinate prediction network. Detect 3D track lines using a 3D track line detection model.

[0009] Optionally, the steps for performing stereo matching and depth calculation based on the image pairs to obtain a depth image are as follows: using a semi-global block matching algorithm to calculate the disparity map between the image pairs; and converting the disparity map into an absolute depth image according to the camera calibration parameters.

[0010] Optionally, the specific steps for fusing depth features with long and short focal length image features using an attention mechanism to obtain strong representational features of depth perception information are as follows: Using the depth features as the query and the telephoto and short-focus features as the key and value, a strong representation feature of the depth perception information is generated through cross-attention calculation.

[0011] Optionally, the specific steps for establishing a 3D track line recognition track line coordinate prediction network based on the strong representational features of the depth-sensing information are as follows: The initialized 3D spatial anchor points are projected onto the strong representational features of the depth-sensing information; By using the back projection matrix composed of camera calibration parameters, the 2D anchor point projection position is obtained, and the geometric relationship between 3D space and 2D features is established. Anchor point feature extraction and offset prediction are performed based on 2D anchor point projection positions. The 3D track line is obtained based on the initial coordinates and predicted offset of the 3D spatial anchor point.

[0012] Optionally, the 3D spatial anchor points are equidistantly distributed along the train's direction of travel in the world coordinate system.

[0013] Optionally, the 3D track line recognition track line coordinate prediction network includes multiple prediction branches, which perform coordinate prediction based on short focal length features, long focal length features and the deep fusion features respectively, and construct a multi-task loss function containing the corresponding branch losses for joint training.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: 1. The method of the present invention constructs a feature fusion and depth estimation mechanism based on multi-focal distance vision. Through the collaborative work of long and short focal length cameras, it realizes direct and accurate mapping from two-dimensional image sequences to three-dimensional orbital space, breaking through the limitations of traditional monocular vision in depth perception.

[0015] 2. The method of this invention designs a track feature extraction and 3D reconstruction scheme with environmental adaptability, which can maintain continuous detection and spatial consistency of the 3D track line under complex scenarios such as changes in lighting, weather interference, and multi-track intersections, significantly improving the robustness and practicality of the system. By achieving the above objectives, this invention aims to establish an efficient and reliable 3D track structure perception system, providing key technical support for automatic train operation and safety. Attached Figure Description

[0016] Figure 1 This is a flowchart of the 3D track line detection method based on the fusion of long and short focal length cameras of the present invention; Figure 2 This is a flowchart of the long and short focal length image feature fusion process in the 3D orbital line detection method based on long and short focal length camera fusion of the present invention. Detailed Implementation

[0017] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0018] One embodiment of the present invention is shown below. Figure 1-2 This paper discloses a 3D track line detection method based on long and short focal length camera fusion. This method utilizes a camera sensor mounted on the front of an urban rail train, combining long and short focal length feature fusion and 3D track line recognition to construct a detection network. Figure 1 As shown, the overall network structure of this invention includes four core functional modules: data preparation, long and short focal length feature fusion, 3D track line recognition, and loss function construction.

[0019] The specific steps are as follows: Step 1: Acquire telephoto and short-focus images from dual cameras, and construct a multi-focal-length visual dataset for 3D orbital detection based on the telephoto and short-focus images.

[0020] In existing train environment perception systems, vision-based 3D track detection technology still faces significant challenges. Although some solutions attempt to introduce stereo vision or sensor fusion, in actual railway operation scenarios, due to the lack of effective modeling methods for the unique geometric structure and spatial continuity of track lines, the system struggles to achieve accurate 3D track reconstruction under long-distance, multi-scale conditions.

[0021] Especially in switch areas, curve transition sections, and complex imaging conditions such as strong light, rain, and fog, traditional methods generally suffer from core problems such as unrobust feature representation, lack of spatial context information, and inconsistency between 2D and 3D feature mapping. These technical shortcomings directly lead to limited accuracy in the system's recovery of the track's 3D geometry, significant drift in long-distance detection results, and difficulty in supporting the high-precision perception requirements of automatic train operation for the track space ahead.

[0022] To fundamentally address the aforementioned problems, this invention constructs a multi-focal-length visual dataset specifically for 3D track line detection. This dataset is based on a rigorously calibrated dual-camera acquisition platform with both long and short focal lengths. The system covers straight sections, curves, turnouts, tunnels, and railway scenes under various weather and lighting conditions, providing robust data support for end-to-end 3D track line detection based on long- and short-focal-length fusion.

[0023] Step 2: Fusion of long and short focal length features to obtain strong representational features of depth perception information. ; This invention synergistically leverages the blind-spot-free advantage of short-focus images and the long-range details of long-focus images, while injecting precise geometric depth information, thereby providing robust and information-rich feature representations for subsequent 3D recognition. The specific implementation steps are as follows: First, a depth image is generated based on stereo matching.

[0024] Receive telephoto images with strict spatiotemporal synchronization With short focal length images A semi-global block matching algorithm is used to calculate the disparity map between them. This algorithm effectively overcomes the scale and deformation problems caused by different focal lengths by constructing and optimizing pixel-level cost cubes.

[0025] Subsequently, based on the camera calibration parameters, the disparity map was... Convert to absolute depth image The expression is: in, For telephoto camera focal length, The binocular baseline distance. These are the horizontal and vertical pixel coordinates of the telephoto image. This step correlates the image coordinates with the actual physical scale.

[0026] In obtaining depth images Then, the system extracts depth images in parallel. Multi-source image features enable semantic encoding of images from different sources in a unified feature space, ensuring feature consistency and laying the foundation for subsequent fusion.

[0027] Specifically, a backbone network with shared weights is used to process the telephoto images separately. Short focal length images and depth images Extract high-level feature maps, denoted as telephoto features. Short focal length characteristics and depth features .

[0028] Finally, a cross-attention mechanism is used to achieve feature fusion in deep perception, resulting in strong representational features. This enables the present invention to effectively establish complex relationships between semantics and geometry.

[0029] Specifically, using depth features that contain precise 3D information As a query, long-focus features rich in texture semantics. and short focal length features Attention is calculated using both keys and values, expressed as:

[0030] in, These represent the query matrix, key matrix, and value matrix in the attention mechanism, respectively. , , , Indicates telephoto features and short focal length features The splicing or optional source, that is, simultaneously or in some strategy, utilizes features from telephoto and short-focus cameras; These are learnable projection weights. This mechanism allows the model to dynamically select and enhance semantic features most relevant to the current depth location from both long- and short-focus images based on spatial location, ultimately outputting a strong representation feature that incorporates depth-aware information. This provides a reliable basis for the accurate identification of 3D track lines.

[0031] Step 3: Establish a trajectory coordinate prediction network for 3D trajectory line recognition; This invention can decode rich image features into accurate three-dimensional spatial orbit lines. Its implementation relies on an anchor-based sequential spatial coordinate regression strategy, and the specific steps are as follows: First, initialize 3D spatial anchor points and then project these anchor points onto strong representation features. .

[0032] Specifically, 3D spatial anchor points are modeled as a series of predefined key points in three-dimensional space, equidistantly distributed along the train's direction of travel (Y-axis), expressed as:

[0033] in, A set representing a sequence of 3D spatial anchor points; These represent the X-axis, Y-axis, and Z-axis coordinates of the Nth anchor point in the three-dimensional world coordinate system, respectively. The coordinates are fixed.

[0034] The 3D spatial anchor points cover the possible track distribution area in front of the train.

[0035] Then, the 2D anchor point projection position is obtained through the back projection matrix composed of camera calibration parameters, and the geometric relationship between 3D space and 2D features is established.

[0036] Next, anchor point features are extracted and offsets are predicted based on the 2D anchor point projection positions.

[0037] Specifically, in strong representational features Based on the projected coordinates of the 2D anchor points, a feature vector is precisely extracted at each anchor point's projected position using bilinear interpolation. This feature vector encodes the visual appearance and contextual information of the local region. Subsequently, this feature vector at each anchor point's projected position is input into a detector head composed of fully connected layers to predict the offset of each 3D spatial anchor point from its neighboring real-world trajectory lines along the horizontal X-axis and vertical Z-axis. The expression is: in, Represents a multilayer perceptron. This represents the bilinear interpolation operation. Represents a projection transformation from 3D to 2D; These represent the X-axis and Z-axis offsets in the world coordinate system for the i-th 3D spatial anchor point to be adjusted to the actual orbital position; This represents the coordinates of the i-th 3D spatial anchor point in the world coordinate system. , , ).

[0038] Finally, the final 3D track line is output through coordinate decoding.

[0039] Specifically, after obtaining the predicted offset, the predicted offset is added to the coordinates of the initialized 3D space anchor point, and the final optimized 3D track line key point coordinates are obtained by decoding.

[0040] For each keypoint, the final optimized 3D trajectory keypoint coordinates The expression is: in, These represent the final X-axis, Y-axis, and Z-axis coordinates of the i-th orbital keypoint in the world coordinate system after optimization and adjustment. The values ​​are the same before and after the optimization.

[0041] Understandably, due to the vertical coordinate Since the anchor points are determined during initialization, regression is unnecessary. All decoded keypoints are connected sequentially to form a complete and continuous 3D trajectory line. This method transforms the complex problem of 3D structure prediction into a regression problem of a series of local offsets, significantly reducing the learning difficulty of the model and ensuring the geometric smoothness and spatial accuracy of the output trajectory line.

[0042] Step 4: Construct a loss function for the track line coordinate prediction network, and train the track line coordinate prediction network based on the loss function; construct a 3D track line detection model based on the trained track line coordinate prediction network.

[0043] To effectively supervise the training of the end-to-end network, this invention constructs a multi-task loss function that can comprehensively measure the accuracy of 3D trajectory prediction. This function not only needs to constrain the final output result, but also needs to supervise the intermediate prediction process to ensure the stability and convergence of the network training.

[0044] First, establish a prediction loss structure based on multi-source features.

[0045] Specifically, the prediction loss structure of the orbital coordinate prediction network includes three parallel detection heads, each based on short focal length features. Long-focus features and deep fusion features The expression for 3D trajectory prediction is:

[0046] in, , and These represent the loss terms for short focal length, long focal length, and depth fusion branches, respectively.

[0047] This invention ensures that the trajectory line coordinate prediction network can simultaneously learn and utilize the unique information from images from different sources, and ultimately rely on the optimal prediction that incorporates depth information.

[0048] Secondly, define the core regression loss for each loss term.

[0049] Specifically, the three loss terms mentioned above employ the same loss calculation method, all based on the L1 norm distance between the predicted and true coordinates, and all focusing on the spatial coordinate error of key points. This is illustrated by the loss from the deep fusion branch. For example, the expression is:

[0050] in, This indicates the total number of track lines that need to be detected in the current processing frame; This represents the initial X-axis coordinate of the k-th anchor point (or key point) on the j-th track before adjustment; This represents the initial Z-axis coordinate of the k-th anchor point (or key point) on the j-th track before adjustment; This represents the number of track lines in the image. Represents the number of key points on each track line; They represent the first Track number The true horizontal and vertical coordinates of each key point, and These are the coordinates of the key points of the final optimized 3D trajectory line. L1 loss is less sensitive to outliers, which helps improve the robustness of the model.

[0051] Finally, a complete optimization objective is formed by weighted summation.

[0052] Specifically, different weights are applied to the prediction errors of different branches or different distance ranges, and weighting coefficients are introduced. Expand the total loss for:

[0053] in, , and These represent the 3D coordinate regression losses generated by predictions based on short-focus image features, long-focus image features, and deep fusion features, respectively.

[0054] During training, the total loss function is minimized through the backpropagation algorithm, which forces the orbital coordinate prediction network not only to accurately predict orbital lines based on various viewpoints, but more importantly, to learn to use deep fusion features to optimize and correct the initial predictions, thereby ultimately outputting orbital lines with high geometric accuracy and good consistency in three-dimensional space.

[0055] Step 5: Use the 3D track detection model to detect the 3D track line.

[0056] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A 3D track line detection method based on long and short focal length camera fusion, characterized in that, The specific steps are as follows: Acquire image pairs, including telephoto and short-focus images, from a synchronously calibrated telephoto camera and a short-focus camera. Based on the image pairs, stereo matching and depth calculation are performed to obtain a depth image; Features of the telephoto image, short-focus image, and depth image are extracted, and the depth features are fused with the features of the telephoto and short-focus images using an attention mechanism to obtain strong representation features of depth perception information. Based on the strong representational features of the depth perception information, a trajectory coordinate prediction network for 3D trajectory line recognition is established. Construct a loss function for the orbital line coordinate prediction network, and train the orbital line coordinate prediction network based on the loss function; then construct a 3D orbital line detection model based on the trained orbital line coordinate prediction network. Detect 3D track lines using a 3D track line detection model.

2. The method according to claim 1, characterized in that, The steps for performing stereo matching and depth calculation based on the image pairs to obtain a depth image are as follows: calculate the disparity map between the image pairs using a semi-global block matching algorithm; and convert the disparity map into an absolute depth image based on the camera calibration parameters.

3. The method according to claim 1, characterized in that, The specific steps for fusing depth features with long and short focal length image features using an attention mechanism to obtain strong representational features of depth perception information are as follows: Using the depth features as the query and the telephoto and short-focus features as the key and value, a strong representation feature of the depth perception information is generated through cross-attention calculation.

4. The method according to claim 1 or 2, characterized in that, Based on the strong representational features of the depth-sensing information, the specific steps for establishing a trajectory coordinate prediction network for 3D trajectory line recognition are as follows: The initialized 3D spatial anchor points are projected onto the strong representational features of the depth-sensing information; By using the back projection matrix composed of camera calibration parameters, the 2D anchor point projection position is obtained, and the geometric relationship between 3D space and 2D features is established. Anchor point feature extraction and offset prediction based on 2D anchor point projection position; The 3D track line is obtained based on the initial coordinates and predicted offset of the 3D spatial anchor point.

5. The method according to claim 4, characterized in that, The 3D spatial anchor points are equidistantly distributed along the train's direction of travel in the world coordinate system.

6. The method according to claim 1, characterized in that, The 3D track line recognition track line coordinate prediction network includes multiple prediction branches, which perform coordinate prediction based on short focal length features, long focal length features and the deep fusion features respectively, and construct a multi-task loss function containing the corresponding branch loss for joint training.

Images