Multi-sight-distance deep learning track line detection method based on global features

Abstract

The invention discloses a multi-sight-distance deep learning track line detection method based on global features, and the method comprises the steps: 1, installing a far-focus camera and a near-focus camera at the head or side surface of a train, obtaining track line images in different scenes in the train operation process, and constructing a data set; 2, in image preprocessing, a Sobel operator is used for calculating gradient energy to generate a focusing degree image, a decision diagram is generated through multi-scale morphological optimization operation, and smooth transition of fused image boundaries is achieved through Gaussian weighted fusion; step 3, constructing an improved ResNet18 backbone network, fusing multi-scale features, combining row selection and structure perception loss, and performing Adam optimization training; and 4, real-time track line detection is carried out. According to the invention, through multi-sight-distance image fusion and global feature extraction, technical support is provided for realizing high-precision and real-time detection of a train track line, automatic train driving, track state monitoring and other key tasks.

Description

Technical Field

[0001] This invention belongs to the field of active safety technology for rail transit, specifically relating to a multi-view deep learning track line detection method based on global features; applied to track line detection in train operation safety monitoring scenarios. Background Technology

[0002] In the train operation safety assurance system, track line detection is one of the core technologies. Its detection accuracy and real-time performance directly affect the decision-making accuracy of the train's automatic driving system and the efficiency of track maintenance. Traditional track line detection technologies often rely on images captured by a single camera (near-focus or far-focus). This results in limited coverage of distant track lines or difficulty in clearly presenting the edges, textures, and other details of nearby track lines. In complex scenarios such as track intersections and switches, false detections or incomplete detection are prone to occur. Furthermore, traditional image fusion methods do not consider the differences between focused and blurred areas in track images, easily leading to blurred edges and loss of details in the fused image. While existing pixel-level segmentation-based track line detection models can achieve high accuracy, their high computational complexity and large number of parameters make them difficult to meet real-time requirements on embedded devices.

[0003] In existing technologies, the SCNN (Spatial Convolutional Neural Network) algorithm enables pixels to communicate with each other and infer occluded parts through long-distance spatial dependencies. However, it still has some significant drawbacks in practical applications, which may limit its performance and efficiency in track line detection tasks:

[0004] Single-line-of-sight detection has significant limitations.

[0005] Current technologies mostly rely on images captured by a single camera (near-focus or far-focus) for detection. While near-focus cameras can clearly capture the detailed features of nearby track lines, their coverage of distant track lines is limited, and they are prone to missing distant track lines due to insufficient image resolution. Far-focus cameras, on the other hand, can cover a wider area of ​​distant track lines, but they struggle to clearly present the edges, textures, and other details of nearby track lines. In complex scenarios such as track intersections and switches, they are prone to false detections or incomplete detection.

[0006] Poor image preprocessing results

[0007] If the image characteristics of the track scene are not optimized, or if traditional image fusion methods (such as simple pixel averaging fusion) are used, the difference between the focused and blurred areas in the track image is not considered. This can easily lead to blurred edges and loss of details in the fused image, which in turn affects the accuracy of the subsequent detection model in extracting track line features.

[0008] Detection model performance imbalance

[0009] On the one hand, while pixel-level segmentation-based track detection models can achieve high accuracy, they are computationally complex and have a large number of parameters, making it difficult to meet real-time requirements on embedded devices. On the other hand, while lightweight detection models can reduce computational costs, they rely heavily on local feature extraction and lack awareness of the global geometric structure of the track. In complex scenarios such as extreme lighting (e.g., strong light, tunnel shadows) and track occlusion (e.g., gravel, fallen leaves), their detection robustness is poor, and they are prone to problems such as track breakage and false detections. Summary of the Invention

[0010] To address the aforementioned issues, this invention discloses a multi-viewpoint deep learning-based track line detection method based on global features. By integrating multi-viewpoint images and extracting global features, it enables high-precision, real-time detection of train track lines, providing technical support for key tasks such as automatic train operation and track condition monitoring.

[0011] To achieve the above objectives, the technical solution of the present invention is as follows:

[0012] A multi-viewpoint deep learning-based trajectory line detection method based on global features includes:

[0013] Step 1: Install telephoto and close-focus cameras on the front or side of the train to acquire track images under different scenarios during train operation and build a dataset;

[0014] Step 2: Image preprocessing uses the Sobel operator to calculate gradient energy to generate a focus map, multi-scale morphological optimization operations to generate a decision map, and Gaussian weighted fusion to achieve a smooth transition of the fused image boundary;

[0015] Step 3: Construct an improved ResNet18 backbone network, integrate multi-scale features, combine row selection and structure-aware loss, and optimize training using Adam;

[0016] Step 4: Real-time track line detection.

[0017] Furthermore, step 1 is detailed as follows:

[0018] (a) Install a telephoto camera and a near-focus camera at the head or side of the train to ensure that the optical axes of the two cameras are parallel and that there is an overlapping area in their fields of view. Collect track images under different scenarios during train operation. During the collection process, use camera calibration and image registration technology to ensure that the images collected by the telephoto camera and the near-focus camera meet the spatial alignment conditions and form paired images.

[0019] (b) The acquired paired images are labeled. The LabelMe labeling tool is used to mark the contour information of the track line in the image, generating remote focal length image samples, near focal length image samples and corresponding paired samples with track line labels. Finally, the processed samples are divided into training set, validation set and test set in a ratio of 7:2:1 to construct the training dataset for track line detection.

[0020] Furthermore, step 2 is detailed below:

[0021] (a) The generation of the focus map is based on the calculation of local gradient energy. The far-focus and near-focus RGB images are converted into grayscale images, and gradient calculation is performed on them. The gradients in the x and y directions are calculated by the 3×3 Sobel operator, and the sum of the squares of the gradients is calculated. The focus map is obtained by summing the gradients in local regions. The gradient energy of the clear focus area is large, and the gradient energy of the blurry area is small.

[0022] (b) The generation of the decision map combines multi-scale morphological optimization operations. The focus maps of the far-focus image and the near-focus image are compared pixel by pixel to generate an initial decision mask. A pixel value of 1 indicates that the far-focus image is clearer at that position, and a pixel value of 0 indicates that the near-focus image is clearer. Then, multi-scale morphological optimization is performed on the initial decision mask. Small-scale structuring elements are used to perform morphological opening operations to eliminate isolated noise points, and large-scale structuring elements are used to perform morphological closing operations to connect broken regions.

[0023] (c) Fusion and boundary smoothing adopts Gaussian weighted fusion method, normalizes the optimized decision map to the range of [0,1] as the initial weight map, performs Gaussian filtering on the weight map to generate a smooth weight map, realizes the smooth transition of the boundary, and then calculates the fused image based on the smooth weight map through the weighted fusion formula.

[0024] Furthermore, step 3 is detailed below:

[0025] (a) The backbone network adopts an improved ResNet18 network architecture, removing the fully connected layers of the original ResNet18 and retaining the convolutional and residual block structures. It is divided into one initial convolutional layer and four residual block stages. Each residual block contains two 3×3 convolutional layers. Skip connections are used to alleviate the gradient vanishing problem. Finally, multi-scale feature maps are output, where low-dimensional features capture details such as edges and textures, and high-dimensional features capture the semantic and geometric structure information of the orbital lines.

[0026] (b) The multi-scale feature fusion module adopts a top-down feature fusion strategy, upsampling the feature maps output from different stages of the backbone network to unify them to the same spatial size. The number of channels in each feature map is unified to 256 through 1×1 convolution. The processed multi-scale feature maps are fused element by element. Global average pooling is applied to compress the fused high-dimensional features into a 512-dimensional global feature vector. Key features are further refined through fully connected layers and activation functions to enhance global context information.

[0027] (c) The row selection module locates the track line based on global features. It predefines h horizontal row anchor points on the input image and distributes them evenly in the height direction of the image. For each row anchor point, the horizontal direction is divided into w grid cells. Each cell corresponds to a candidate track point. By combining the global feature vector with row-specific features, the probability of each grid cell in each row containing a track point is predicted through a classification network. The prediction results of each row are post-processed (such as non-maximum suppression) to determine the most likely track point position in that row. The track points of all rows are connected to form a complete track line.

[0028] (d) The structure perception loss module adopts a combined loss function to optimize the model. The classification loss uses cross-entropy loss to optimize the classification results of track points in each row of grid cells. The structure loss uses a second-order difference equation to model the shape of the lane and a similarity loss function is designed to model the positional relationship between adjacent track points.

[0029] (e) The model was trained using the track line dataset constructed in step S1. The dataset contains track images and corresponding track line annotations under different scenes and lighting conditions. The optimizer was Adam, with an initial learning rate of 1e4. The learning rate was adjusted using a cosine annealing strategy, and the batch size was set to 16. The model was trained iteratively until it converged (the validation set loss did not decrease significantly for 5 consecutive epochs).

[0030] The beneficial effects of this invention are as follows:

[0031] (1) This invention utilizes paired image acquisition from telephoto and near-photo cameras to cover multiple observation scales (long-distance global structure + close-range details), effectively addressing imaging differences of the orbital line at different distances and improving data diversity and robustness. LabelMe is used to finely annotate the orbital line contour, constructing a high-quality labeled dataset, and the training / validation / test sets are divided in a 7:2:1 ratio to ensure sufficient model training, reliable evaluation, and improved generalization ability. Images covering different weather conditions, lighting, speeds, and orbital states are collected, enabling the model to adapt to complex operating environments and possess strong environmental adaptability.

[0032] (2) This invention employs a lightweight multi-focus image fusion method, effectively fusing detailed features of near-field track lines with global information of far-field track lines, avoiding the missed detection problem caused by insufficient resolution and coverage of single-view images. A focus map is generated by calculating local gradient energy using the Sobel operator, accurately identifying the clear regions in both near and far-field images, suppressing blur and noise interference, and improving the signal-to-noise ratio of subsequent detection. Multi-scale morphological operations (small-scale opening operation for denoising + large-scale closing operation for connectivity) are introduced to generate a structurally complete decision map, optimizing the continuity of decision boundaries. A Gaussian weighted fusion strategy is used to filter the decision map, avoiding fusion artifacts caused by hard boundaries, generating a visually natural, edge-continuous, high-quality fused image, providing better input for track detection.

[0033] (3) The row selection track detection model based on global image features takes into account both details and semantic features. Its backbone network adopts the ResNet18 network architecture, removes the fully connected layers of the original network, and retains the multi-stage residual structure. Low-level features capture track edges and texture details, while high-level features extract semantic and geometric structures, forming complementary representations. The multi-scale feature fusion module adopts a top-down feature fusion strategy, fusing feature maps from different stages of the backbone network. After unifying the number of channels through 1×1 convolution, the feature maps are superimposed to solve the detection limitations of single-scale features. Then, global average pooling is used to compress high-dimensional features into global vectors, enhancing multi-scale context awareness. The row selection module with row anchor point + grid classification mechanism uses global features to transform track detection into a row-by-row localization problem. The number of predefined row anchor points and grid units is much smaller than the image size, greatly reducing the computational scale. The structure-aware loss module adopts a combination loss function of "classification loss + structure loss". Through these loss functions, the aim is to model the positional relationship between track points, effectively preventing jitter, breakage, or unreasonable bending in the detection results. Attached Figure Description

[0034] Figure 1 This is a structural diagram of the track line detection model.

[0035] Figure 2 A visual annotation diagram for track line detection. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0037] The software environment for the multi-viewpoint deep learning trajectory line detection method based on global features described in this invention is as follows:

[0038] Operating System: The annotation and training equipment uses Windows 10.

[0039] Development framework: Python 3.8, PyTorch 1.12.0, OpenCV 4.6.0;

[0040] Labeling tool: LabelMe 5.1.1.

[0041] The specific testing methods are as follows:

[0042] Step 1: Data Acquisition and Dataset Construction

[0043] (a) Install a telephoto camera (200mm focal length) and a near-focus camera (35mm focal length) at the head or side of the train to ensure that the optical axes of the two cameras are parallel and that there is an overlapping area in their fields of view. Collect track images under different scenarios (straight track, curved track, turnout, tunnel, strong light, rain, etc.) during the train operation. During the acquisition process, use camera calibration and image registration technology to ensure that the images collected by the telephoto camera and the near-focus camera meet the spatial alignment conditions and form paired images.

[0044] (b) The acquired paired images are labeled. The LabelMe labeling tool is used to mark the contour information of the track line in the image, generating remote focal length image samples, near focal length image samples and corresponding paired samples with track line labels. Finally, the processed samples are divided into training set, validation set and test set in a ratio of 7:2:1 to construct the training dataset for track line detection.

[0045] Step 2: Image Preprocessing

[0046] (a) Generation of focus map

[0047] RGB to grayscale image conversion uses the standard luminance conversion formula for grayscale value calculation:

[0048] ;

[0049] in , , For input RGB image in pixels The three-channel values ​​at that location, To output grayscale values, the above formula is applied to each pixel, thus converting the telephoto image... and close-focus images Convert to grayscale images respectively and .

[0050] The gradients in the x and y directions are calculated using the 3×3 Sobel operator.

[0051] Oriented gradient (horizontal edge detection):

[0052] ;

[0053] Oriented gradient (vertical edge detection):

[0054] ;

[0055] Sum of squared gradients:

[0056] ;

[0057] Where * denotes the convolution operation, and E(x,y) is the gradient energy for grayscale images. and Calculate the gradient energy map separately and .

[0058] Focusability is defined as the sum of gradient energy within a 5×5 window:

[0059] ;

[0060] in The focus value is calculated using a 5x5 window. Padding is used to ensure the validity of boundary pixel calculations. Box filtering is used for parallel calculation and local summation to generate a telephoto focus map. and near-focus focus map .

[0061] (b) Decision graph generation

[0062] Pixel-by-pixel comparison of focus maps to generate a binary decision mask:

[0063] ;

[0064] in This indicates that the pixel selects a telephoto image. This indicates that a near-focus image is selected.

[0065] Then, multi-scale morphological optimization is performed on the initial decision mask, using small-scale structuring elements for morphological opening operations to eliminate isolated noise points:

[0066] ;

[0067] Connect fracture regions using morphological closing operations with large-scale structuring elements:

[0068] ;

[0069] in It is a 3×3 elliptical structural element. Given a 5×5 elliptical structuring element, perform opening and closing operations sequentially to output the optimized decision graph. .

[0070] (c) Blending and Boundary Smoothing

[0071] The decision graph is smoothed by Gaussian convolution, and a weight graph W is generated.

[0072] Gaussian kernel definition (15×15 window, ):

[0073] ;

[0074] Weighted graph calculation:

[0075] ;

[0076] in This is the smoothed weighted graph, with values ​​ranging from [0,1].

[0077] Execute the weighted fusion formula and output the final fused image. The pixel values ​​of the fused image are a weighted sum of those of the telephoto and near-photo images:

[0078] ;

[0079] in To merge the RGB values ​​of the image at pixel (x,y).

[0080] Step 3: Detection Model Construction and Training

[0081] (a) Row selection modeling based on global features

[0082] By using global features, the correct track position is selected on each predefined row. The track line is represented as the horizontal coordinate at the predefined row anchors. At each row anchor, the image width is divided into multiple cells, and track line detection can be described as selecting a specific cell at each predefined row anchor. Assuming the maximum number of tracks is C, the number of row anchors is h, and the number of grid cells is w, let X represent the global image features. This is a classifier used to select the orbital position at the j-th row anchor point on the i-th orbit. The orbital prediction result can be expressed as:

[0083] ;

[0084] in, It is a (w+1)-dimensional vector (an extra dimension is added to represent the case where the track line does not exist), representing the selection probability of the (w+1) grid cells corresponding to the i-th track line and the j-th row anchor point.

[0085] set up This is the correct position (i.e., the grid cell where the actual trajectory line is located). Representing cross-entropy loss, the optimization process of this modeling method can be expressed as:

[0086] ;

[0087] (b) Track structure loss

[0088] The position of the track is represented by a classification vector, so the continuity of the track can be achieved by constraining the distribution of classification vectors at adjacent row anchor points. The first structural loss function designed is the similarity loss function. This is used to measure the similarity of classification vectors between adjacent rows, encouraging the model to learn the continuity features of the track.

[0089] ;

[0090] in, This represents the prediction result for the i-th orbital line and the j-th row anchor point, where ||·||1 represents the L1 norm (i.e., the sum of the absolute values ​​of the vector elements).

[0091] The second structural loss function focuses on the shape characteristics of the orbital track, using the expected value of the prediction results to approximate the orbital position, that is, using the softmax function to obtain the probability distribution of each position:

[0092] ;

[0093] Where w is a w-dimensional vector (the number of grid cells), and This represents the probability of each position.

[0094] This area excludes background mesh cells, and the calculation range is only from 1 to w. The expected location can be expressed as:

[0095] ;

[0096] in, Let represent the probability corresponding to the i-th lane, the j-th row anchor point, and the k-th position. This positioning method has two advantages: first, the expectation function is differentiable; second, this operation can recover continuous positions from discrete random variables, and its second-order difference constraint can be expressed as:

[0097] ;

[0098] in, This indicates the position at the j-th line anchor point in the i-th lane.

[0099] The reason for choosing to use second-order difference instead of first-order difference is that first-order difference is not zero, which means that the network needs additional parameters to learn the distribution of first-order difference of orbit position; secondly, compared with first-order difference, second-order difference has relatively weaker constraint strength, so it has a smaller impact in the scenario of curved orbit.

[0100] Ultimately, the overall structural loss can be expressed as:

[0101] ;

[0102] in, It is the loss coefficient.

[0103] Step 4, Real-time Track Line Detection:

[0104] During train operation, track line images are acquired in real time from both telephoto and near-photo cameras. Following the image preprocessing procedure in step 2, the real-time acquired images are fused to obtain a real-time fused image. This real-time fused image is then input into the track line detection model trained in step 3 for inference, generating the contour information of the track line. A visual annotation layer (using green dotted lines to mark the trajectory) is then overlaid on the fused image.

[0105] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-viewpoint deep learning-based trajectory line detection method based on global features, characterized in that: include: Step 1: Install telephoto and close-focus cameras on the front or side of the train to acquire track images under different scenarios during train operation and build a dataset; Step 2: Image preprocessing uses the Sobel operator to calculate gradient energy to generate a focus map, multi-scale morphological optimization operations to generate a decision map, and Gaussian weighted fusion to achieve a smooth transition of the fused image boundary; Step 3: Construct an improved ResNet18 backbone network, integrate multi-scale features, combine row selection and structure-aware loss, and optimize training using Adam; Step 4: Real-time track line detection.

2. The multi-viewpoint deep learning trajectory detection method based on global features according to claim 1, characterized in that: Step 1 is as follows: (a) Install a telephoto camera and a near-focus camera at the head or side of the train to ensure that the optical axes of the two cameras are parallel and that there is an overlapping area in their fields of view. Collect track images under different scenarios during train operation. During the collection process, use camera calibration and image registration technology to ensure that the images collected by the telephoto camera and the near-focus camera meet the spatial alignment conditions and form paired images. (b) The acquired paired images are labeled. The LabelMe labeling tool is used to mark the contour information of the track line in the image, generating remote focal length image samples, near focal length image samples and corresponding paired samples with track line labels. Finally, the processed samples are divided into training set, validation set and test set in a ratio of 7:2:1 to construct the training dataset for track line detection.

3. The multi-viewpoint deep learning trajectory detection method based on global features according to claim 1, characterized in that: Step 2 is as follows: (a) Generation of the focus map RGB to grayscale image conversion uses the standard luminance conversion formula for grayscale value calculation: ； in , , For input RGB image in pixels The three-channel values ​​at that location, To output grayscale values, the above formula is applied to each pixel, thus converting the telephoto image... and close-focus images Convert to grayscale images respectively and ; The gradients in the x and y directions are calculated using the 3×3 Sobel operator. Oriented gradient, horizontal edge detection: ； Oriented gradient, vertical edge detection: ； Sum of squared gradients: ； Where * denotes the convolution operation. For gradient energy, for grayscale images and Calculate the gradient energy map separately and ; Focusability is defined as the sum of gradient energy within a 5×5 window: ； in The focus value is calculated using a 5x5 window. Padding is used to ensure the validity of boundary pixel calculations. Box filtering is used for parallel calculation and local summation to generate a telephoto focus map. and near-focus focus map ; (b) Decision graph generation Pixel-by-pixel comparison of focus maps to generate a binary decision mask: ； in This indicates that the pixel selects a telephoto image. Indicates selection of a near-focus image; Then, multi-scale morphological optimization is performed on the initial decision mask, using small-scale structuring elements for morphological opening operations to eliminate isolated noise points: ； Connect fracture regions using morphological closing operations with large-scale structuring elements: ； in It is a 3×3 elliptical structural element. Given a 5×5 elliptical structuring element, perform opening and closing operations sequentially to output the optimized decision graph. ; (c) Blending and Boundary Smoothing The decision graph is smoothed using Gaussian convolution, and a weight graph W is generated. Gaussian kernel definition (15×15 window, ): ； Weighted graph calculation: ； in This is the smoothed weighted graph, with values ​​ranging from [0,1]. Execute the weighted fusion formula and output the final fused image. The pixel values ​​of the fused image are a weighted sum of those of the telephoto and near-photo images: ； in To merge the RGB values ​​of the image at pixel (x,y).

4. The multi-viewpoint deep learning trajectory detection method based on global features according to claim 1, characterized in that: Step 3 is as follows: (a) Row selection modeling based on global features By using global features, the correct track position is selected on each predefined row. The track line is represented as the horizontal coordinate at the predefined row anchors. At each row anchor, the image width is divided into multiple cells, and track line detection is described as selecting a specific cell at each predefined row anchor. Assuming the maximum number of tracks is C, the number of row anchors is h, and the number of grid cells is w, let X represent the global image features. It is a classifier used to select the track position at the j-th row anchor point on the i-th track; the track prediction result is expressed as: ； in, It is a (w+1)-dimensional vector, representing the selection probability of (w+1) grid cells corresponding to the i-th track line and the j-th row anchor point; set up For the correct position, Let represent the cross-entropy loss. The optimization process for this modeling method can then be expressed as: ； (b) Track structure loss The position of the track is represented by a classification vector, so the continuity of the track is achieved by constraining the distribution of classification vectors at adjacent row anchor points; the first structural loss function designed is the similarity loss function. This is used to measure the similarity of classification vectors between adjacent rows, encouraging the model to learn the continuity features of the track; ； in, This represents the prediction result for the i-th track line and the j-th row anchor point, where ||·||1 represents the L1 norm; The second structural loss function focuses on the shape characteristics of the orbital track, using the expected value of the prediction results to approximate the orbital position, that is, using the softmax function to obtain the probability distribution of each position: ； Where w is the number of grid cells, and Represents the probability of each position; This area excludes background mesh cells, and the calculation range is only from 1 to w. The expected location is represented as: ； in, Let represent the probability corresponding to the i-th lane, the j-th row anchor point, and the k-th position. This positioning method has two advantages: first, the expectation function is differentiable; second, this operation recovers continuous positions through discrete random variables, and its second-order difference constraint is expressed as: ； in, This indicates the position at the j-th lane anchor point; Ultimately, the overall structural loss is expressed as: ； in, It is the loss coefficient.

5. The multi-viewpoint deep learning trajectory detection method based on global features according to claim 1, characterized in that: Step 4 is as follows: During train operation, the track line images acquired in real time by the telephoto and near-focus cameras are first obtained according to step 1. The images are then fused according to the image preprocessing process in step 2 to obtain a real-time fused image. The real-time fused image is then input into the track line detection model trained in step 3 for inference to generate the contour information of the track line. A visual annotation layer is then superimposed on the fused image.

Images