Ballast bed foreign matter detection method based on feature comparison

Abstract

The invention discloses a ballast bed foreign matter detection method based on feature comparison, and relates to the technical field of ballast bed foreign matter detection.The ballast bed foreign matter detection method comprises the steps that a ballast bed historical image containing a depth map and a grey-scale map and corresponding position information are triggered and collected through a position encoder, and the ballast bed historical image and the corresponding position information are bound to serve as a background image to be stored in a background database; collecting a current ballast bed image, binding the current ballast bed image with corresponding position information, taking the current ballast bed image and the corresponding position information as a foreground image, extracting a foreground initial image from the foreground image based on the position information, and performing pixel-level registration on the foreground initial image and a plurality of background initial images at corresponding positions in the background database; multi-scale feature extraction is carried out on the foreground image and the background image after registration, then the extracted multi-scale features are compared to output multi-scale difference features, finally, the multi-scale difference features are detected, and a ballast bed foreign matter detection result is output and obtained. The problems that existing manual inspection efficiency is low, traditional image processing accuracy is low, and a deep learning method is affected by data long-tail distribution are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of track bed foreign object detection, and in particular to a track bed foreign object detection method based on feature comparison. BACKGROUND

[0002] In recent years, with the rapid development of high-speed railways, more and more people choose railway transportation as their preferred mode of transportation, and higher requirements for the safety and stability of railway operation are put forward. In the actual operation of high-speed railways, scattered parts or foreign abnormal objects are prone to appear near the track bed. The strong airflow generated by the high-speed train may lift these foreign objects, causing the foreign objects to collide with the train, resulting in structural damage to the train and posing a serious safety hazard to the operation of high-speed trains.

[0003] Currently, the methods for detecting track bed foreign objects mainly include manual inspection, traditional image processing-based foreign object detection, and deep neural network-based foreign object detection. However, each method has certain problems. Manual inspection relies on experienced workers to conduct manual inspection and removal along the railway line. This method is not only inefficient and costly, but also susceptible to the experience and working state of the inspectors, resulting in misjudgment and missed judgment. The traditional image processing-based method involves collecting grayscale images of the track bed to build a background model and performing area threshold processing. Then, the foreign object area is obtained through template comparison, connected domain, and morphological processing. Although this method has fast detection speed, it has low accuracy and poor robustness. The deep neural network-based foreign object detection method mostly uses deep learning target detection frameworks such as SSD, FasterR-CNN, YOLOV4, and YOLOV5 for foreign object detection. Due to the small size and diverse types of track bed foreign objects, it is difficult to collect a sufficient number and types of foreign object images, resulting in a serious long-tail distribution phenomenon in the data set and reducing the foreign object detection capability. SUMMARY

[0004] Therefore, the present application provides a track bed foreign object detection method based on feature comparison to solve the problems in the prior art.

[0005] The first aspect of the present application provides a track bed foreign object detection method based on feature comparison, which comprises: Under the trigger of the position encoder, the track bed historical images and the corresponding position information in the line are collected, and the track bed historical images include depth maps and grayscale images; The collected track bed historical images and the corresponding position information are bound as background images and stored in a background database; The current track bed images and the corresponding position information in the line are collected and bound as foreground images, and the starting point image in the foreground image is extracted and recorded as the foreground starting image; The position information of the foreground starting image is recorded as foreground position information, a plurality of background starting images corresponding to positions of the foreground position information are retrieved from the background database, and all the background starting images are stored into a background starting image set; Pixel-level registration is performed on the foreground starting image and all the images in the background starting image set, to obtain a registered foreground image, a registered background image, and a registration relationship, and subsequent foreground images and background images are registered based on the registration relationship; Multi-scale feature extraction is performed on the registered foreground image and the registered background image, to obtain an extraction result; Based on the extraction result, multi-scale difference features are extracted by comparing the multi-scale features in the registered foreground image with the multi-scale features in the registered background image; All the multi-scale difference features are detected, and a ballast foreign matter detection result is output.

[0006] In a possible implementation manner of the first aspect, the multi-scale feature extraction performed on the registered foreground image and the registered background image to obtain an extraction result includes: The registered foreground image and the registered background image are preprocessed to obtain a feature map, including convolution processing and pooling processing; The preprocessed feature map sequentially enters different levels of feature extraction layers, each feature extraction layer processes features in the feature map by depth through a plurality of bottleneck structures and down-samples to generate a plurality of feature maps of different scales; The plurality of feature maps of different scales are packed and integrated to obtain the extraction result.

[0007] In a possible implementation manner of the first aspect, each bottleneck structure includes three layers of convolution, BN batch normalization, and ReLU activation function, and the three layers of convolution include 1x1Conv dimension reduction, 3x3Conv feature extraction, and 1x1Conv dimension increase.

[0008] In a possible implementation manner of the first aspect, the multi-scale difference feature extraction includes: Feature compression is performed on the integrated feature maps of all scales; Feature encoding is performed on the feature maps after feature compression to extract corresponding high-order semantic information; A foreground feature about the foreground image after feature encoding is taken as a first-dimensional feature, a background feature about the background image after feature encoding is taken as a second-dimensional feature, and the first-dimensional feature and the second-dimensional feature are multiplied pixel by pixel as a third-dimensional feature; The first-dimensional feature, the second-dimensional feature, and the third-dimensional feature are superimposed in a channel dimension to output the multi-scale difference feature.

[0009] In one possible implementation of the first aspect, feature compression of all integrated scale feature maps includes: The number of feature channels in feature maps at different scales is compressed three times through convolutional layers; After the first compression, the number of feature channels becomes half the number of input channels plus the number of basic channels; After the second compression, the number of feature channels becomes four times the number of basic channels; After the third compression, the number of feature channels becomes twice the number of basic channels.

[0010] In one possible implementation of the first aspect, all multi-scale difference features are detected, and the output of the foreign object detection results for the track bed includes: Each scale's difference features are processed separately using a three-layer fully connected layer; After three layers of fully connected processing, the scale difference features are fused and encoded across scales. Multi-scale difference features, after inter-scale fusion encoding, are decoded based on the detection head to output foreign objects in the track bed; The output of foreign objects in the track bed is taken as the detection result of foreign objects in the track bed.

[0011] In one possible implementation of the first aspect, inter-scale fusion coding includes: The multi-scale difference features corresponding to multiple feature maps at different scales are denoted as the second-scale difference feature, the third-scale difference feature, and the fourth-scale difference feature, respectively, from high to low resolution. The fourth-scale difference features are subjected to two deconvolution operations to generate the first fusion feature; After performing a deconvolution operation on the fourth-scale difference feature, it is fused with the third-scale difference feature, and then after another deconvolution operation, a third fused feature is generated. After performing a deconvolution operation on the fourth-scale difference feature, it is fused with the third-scale difference feature. After another deconvolution operation, it is fused with the second-scale difference feature to generate the second fused feature. The fourth-scale difference feature is deconvolved twice, and the third-scale difference feature is deconvolved once. After fusing with the second-scale difference feature, a convolution operation is performed to generate the first fused feature. The first fusion feature, the second fusion feature, the third fusion feature, and the fourth fusion feature are superimposed on the channel dimension to complete the inter-scale fusion coding.

[0012] In one possible implementation of the first aspect, the multi-scale difference features after inter-scale fusion encoding are decoded based on the detection head, including: The multi-scale difference features encoded by inter-scale fusion are sequentially processed by 1×1 convolution, deconvolution, 3×3 convolution, and Sigmoid activation function to complete feature decoding and upsampling, thereby obtaining the foreign objects in the track bed and completing the detection of foreign objects in the track bed.

[0013] In one possible implementation of the first aspect, pixel-level registration of the depth map and grayscale map of the foreground starting image and all background starting images respectively includes: Obtain the grayscale image and depth image corresponding to the foreground starting image, and denot them as the first grayscale image and the first depth image, respectively. Obtain the grayscale and depth maps corresponding to all background starting images, and denote them as the second grayscale image and the second depth map, respectively. Feature points are extracted from the first grayscale image and all second grayscale images using a feature extraction algorithm. The similarity between the feature points in the first grayscale image and all second grayscale images is calculated using feature descriptors, and similar feature pairs are selected. Based on the first depth map and all second depth maps, extract the depth values ​​of each pair of feature points in the corresponding depth map, calculate the corresponding depth difference, and determine whether the depth difference is less than a preset threshold. If it is, it is determined to be a real feature pair; otherwise, it is determined to be a false feature pair. Based on the three-dimensional information of the first depth map and all second depth maps, extract the spatial coordinates of each pair of feature points in the real feature pair, and verify whether the spatial geometric relationship of each pair of feature points in the real feature pair is consistent. If yes, no action is taken; otherwise, the corresponding real feature pair is cleaned. Based on the cleaned real feature pairs, a spatial transformation model between the foreground and background images is constructed. Based on the spatial transformation model, the pixels of the background starting image are mapped to the coordinate system of the foreground starting image, thereby achieving pixel-level alignment between the foreground starting image and the background starting image.

[0014] One possible implementation of the first aspect also includes: Multiple starting points are set for the line, and a detection interval is formed between any adjacent starting points. All starting points divide the line into multiple detection intervals, and foreign object detection of the track bed is performed in each detection interval.

[0015] The beneficial effects are as follows: This invention discloses a method for detecting foreign objects in roadbeds based on feature comparison. Triggered by a position encoder, it acquires historical images of the roadbed, including depth and grayscale images, along with corresponding position information. These images are then bound and stored as background images in a background database. The current roadbed image and its corresponding position information are acquired and bound as foreground images. A foreground starting image is extracted from the foreground image based on its position information and pixel-level registered with multiple background starting images at the corresponding positions in the background database. Multi-scale feature extraction is performed on the registered foreground and background images. The extracted multi-scale features are then compared to output multi-scale difference features. Finally, the multi-scale difference features are detected, and the result of detecting foreign objects in the roadbed is obtained. This invention solves the problems of low efficiency in existing manual inspections, low accuracy in traditional image processing, and the influence of long-tailed data distribution on deep learning methods. It not only improves the accuracy of foreign object detection in roadbeds but also adapts to complex environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a diagram illustrating a method for detecting foreign objects in a track bed based on feature comparison, provided in an embodiment of this application. Figure 2 This is a schematic diagram of the multi-scale feature extraction process provided in the embodiments of this application; Figure 3 This is a schematic diagram of the process for comparing and outputting multi-scale difference features according to an embodiment of this application; Figure 4 This is a schematic diagram of the process for detecting multi-scale difference features provided in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0020] Example Currently, the main methods for detecting foreign objects on the track bed include manual inspection, traditional image processing, and deep neural network-based foreign object detection. Traditional manual inspection suffers from low efficiency, high cost, and a high rate of false positives and false negatives; traditional image processing suffers from low detection accuracy and poor robustness; and deep neural network-based foreign object detection suffers from low detection capability due to the small size and variety of foreign objects on the track bed, resulting in a long-tailed distribution of the dataset.

[0021] Therefore, this application provides a method for detecting foreign objects in roadbeds based on feature comparison, such as... Figure 1 As shown, it includes: Triggered by the position encoder, historical images of the track bed and corresponding position information are acquired. The historical images of the track bed include depth maps and grayscale images. The collected historical images of the track bed and their corresponding location information are bound together and used as background images, which are then stored in the background database. The current track bed image and corresponding position information in the line are collected and bound as the foreground image, and the starting point image in the foreground image is extracted and recorded as the foreground starting image; The position information of the foreground starting image is obtained and recorded as the foreground position information. Multiple background starting images corresponding to the position of the foreground position information are retrieved from the background database, and all background starting images are stored in the background starting image set. Pixel-level registration is performed on the foreground starting image and all images in the background starting image set to obtain the registered foreground image, background image and registration relationship. Subsequent foreground images and background images are registered based on the registration relationship. Multi-scale feature extraction is performed on the registered foreground and background images to obtain the extraction results; Based on the extraction results, the multi-scale features in the registered foreground image and the multi-scale features in the registered background image are compared to extract multi-scale difference features. All multi-scale differential features are detected, and the results of foreign object detection in the track bed are output.

[0022] This embodiment provides a method for detecting foreign objects in a track bed based on feature comparison. Its workflow includes: 1. Background database construction: Triggered by the position encoder, the system begins acquiring historical images of the track bed and their corresponding location information. The acquired historical track bed images and their corresponding location information are then bound together as background images and stored in the background database. 2. Acquisition of foreground and starting point images: The system collects real-time images of the current track bed and its corresponding location information, binds them, and uses them as the foreground image. It also extracts the starting point image from the foreground image and records it as the foreground starting image, while recording the corresponding location information as the foreground location information.

[0023] 3. Starting point background image retrieval: Based on the foreground position information of the foreground starting image, multiple starting point background images are retrieved from the background database at the corresponding positions and recorded as the background starting images to ensure the consistency of the positions of the comparison objects.

[0024] 4. Pixel-level registration processing: Pixel-level registration is performed on the foreground starting image and all background starting images. Essentially, this utilizes feature matching of grayscale images and spatial location constraints of depth maps to achieve dual-modal collaborative calibration, specifically including: For the grayscale images of the starting point images corresponding to the foreground image and the starting point images corresponding to the background image, feature extraction algorithms (such as SIFT and SURF) are used to extract stable texture feature points. Feature points with inherent structure of the track bed (such as track edge corners, corners of ballast particles, and intersections of ballastless track bed slab seams) are selected first. The positions of these feature points in the foreground and background images are relatively fixed, avoiding the extraction of unstable feature points such as stains and reflections.

[0025] The similarity of feature points extracted from the foreground and background grayscale images is calculated using feature descriptors. Similar feature pairs, such as those with a Euclidean distance less than a threshold, are selected to initially establish the pixel correspondence between the foreground and background images (e.g., the intersection point A of the track slab seam in the first grayscale image corresponds to the same intersection point A of the slab seam in the second grayscale image). The aim is to obtain a large number of reliable pixel correspondences between the foreground and background images through the texture consistency of the grayscale images, providing a foundation for subsequent spatial transformations.

[0026] For each pair of feature points matched in the similar feature pairs, extract the depth values ​​of the two in the corresponding depth map and calculate the corresponding depth difference. If the depth difference is less than a preset threshold, the similar feature pairs are determined to be real feature pairs. If the depth difference is greater than or equal to the preset threshold, the similar feature pairs are determined to be false feature pairs and are removed.

[0027] Based on the 3D information of the first and second depth maps, the spatial coordinates of each pair of feature points in the true feature pair are calculated (the depth values ​​are converted into XYZ coordinates in the world coordinate system based on the camera intrinsics), and the consistency of the spatial geometric relationship of the matching point pairs is verified, thereby further cleaning up false matches caused by local texture similarity.

[0028] Based on the cleaned real feature pairs, a spatial transformation model (such as affine transformation or perspective transformation) is constructed between the foreground image and the background image. The spatial transformation model maps the coordinate system of the background starting image in the background image to the coordinate system of the foreground starting image in the foreground image, ultimately achieving pixel-level alignment between the foreground image and the background image.

[0029] 5. Multi-scale feature extraction: Please refer to Figure 2 The process begins with preprocessing. The input foreground and background images undergo preprocessing, including convolution and pooling. Local features are extracted using a Conv7×7 sliding convolution kernel, increasing the original image's channel count to 64. This channel increase essentially expands the feature dimension, capturing more local features. Max pooling downsamples the convolutional feature map using a fixed-size pooling window (e.g., 2×2, stride 2), reducing the image resolution to 1 / 4 of the original. The core purpose of downsampling is to reduce pixel redundancy, decreasing subsequent computation while preserving key features by extracting local maxima, avoiding interference from irrelevant details. The final output is a 64-channel, 1 / 4 resolution feature map, providing a unified input data format for subsequent multi-stage feature extraction at multiple scales.

[0030] The multi-stage extraction process (stage 1-stage 4) involves preprocessing the feature maps into four independent feature extraction stages, also known as feature extraction layers. Each feature extraction layer uses multiple bottleneck structures to perform in-depth processing and downsampling on the features, ultimately generating four feature maps at different scales (Scale 1-Scale 4) (decreasing resolution, increasing number of channels). The essence of the scale difference is that after each feature extraction layer, downsampling further reduces the resolution while increasing the number of channels.

[0031] Each feature extraction layer contains multiple bottleneck structures. Each bottleneck structure consists of three convolutional layers: "1×1Conv→3×3Conv→1×1Conv" + Batch Normalization (BN) + ReLU (activation function). The core logic is to first compress the number of channels to reduce computation, then extract features, and finally restore the number of channels to maintain feature dimensionality. Specifically: the number of channels in the input feature map is first compressed to 1 / 4 of the original number using 1×1Conv; then, 3×3Conv is used to convolve the dimensionality-reduced feature map to extract key local features of the track bed and foreign objects. Simultaneously, BN and ReLU are immediately followed by convolution; the number of channels in the output feature map after convolution is restored to the same number as the number of channels when inputting the bottleneck structure using 1×1Conv. The purpose of dimensionality enhancement is to ensure the continuity of feature dimensionality. The output of each bottleneck structure is superimposed on the input of the original bottleneck structure through an "Add" operation to form residual connections.

[0032] The four feature maps at different scales are divided into four functional groups: the low-scale (Scale 1, high resolution) focuses on detailed features (such as the texture and edges of small foreign objects), the high-scale (Scale 4, low resolution) focuses on semantic features (such as the overall shape of the foreign object and its spatial relationship with the track bed), and the intermediate scales (Scale 2-Scale 3) facilitate the transition from details to semantics, ensuring that features of foreign objects of different sizes and shapes can be captured. Since the Scale 1 low-scale features have high resolution but low semantic value, they can be covered by indirectly fusing other scales, so they do not need to participate in subsequent integration. Finally, the multi-scale feature maps output from the three feature extraction layers (Scale 2-Scale 4) are packaged and integrated to obtain the extraction result.

[0033] 6. Multi-scale differential feature extraction: Please refer to Figure 3 The convolutional layers compress the feature channel counts of the foreground (ForeFeatureScale) and background (BackFeatureScale) multi-scale feature maps three times: the first compression is half the number of input channels plus the number of basic channels; the second compression is four times the number of basic channels; and the third compression is twice the number of basic channels. Each compression step involves both "ConV1×1+BN+ReLU" and "ConV3×3+BN+ReLU". The purpose of this compression is to unify the feature channel dimensions, preventing calculation errors caused by dimensionality confusion during subsequent encoding; to remove redundant features, retain core information, and improve the expressive power of key features; and to balance the weights of features at different scales.

[0034] While compressed features address the dimensionality issue, they still primarily consist of local, low-order features, making it difficult to distinguish between similar features such as track bed stains and small foreign objects. Therefore, feature encoding is necessary. The essence of feature encoding is to extract high-order semantic information. By stacking convolutional layers and activation functions, the compressed features are processed deeply, integrating local low-order features into global high-order features. This enhances the discriminative power of the features, making the features of the inherent structure of the track bed in the foreground and background images more consistent, while highlighting the unique characteristics of foreign objects.

[0035] The encoded foreground and background features already possess high discriminative power. Through three-dimensional feature construction and channel concat overlay, accurate capture of the differences between the foreground and background images is achieved. The encoded foreground features, containing complete semantic information of "track bed structure + foreign object" in the foreground image, are directly retained as the first-dimensional feature cat1. The encoded background features, containing only the semantic information corresponding to the track bed structure without foreign objects, are directly retained as the second-dimensional feature cat2. The foreground and background features are multiplied pixel by pixel, and the difference between them is amplified through multiplication to generate the third-dimensional feature cat3. cat1, cat2, and cat3 are then overlaid along the channel dimension (concat) to form a multi-dimensional difference feature map. This combination of direct features and difference amplification features comprehensively preserves the difference information between the foreground and background images, avoiding subtle differences that might be missed in single-dimensional comparisons, while amplifying key differences through cat3 to ensure accurate location of foreign objects in subsequent foreign object detection.

[0036] 7. Foreign Object Detection and Result Output: Please refer to Figure 4 The core principle is intra-scale feature enhancement + multi-scale feature fusion + precise decoding processing. By performing "deep integration - semantic parsing - spatial restoration" on the multi-scale difference features of the preceding output, a foreign object mask of the same size as the original input image is finally generated, realizing the precise location and detection of foreign objects.

[0037] First, there's the intra-scale channel interaction. The multi-scale difference features output earlier (corresponding to Scale1-Scale4) already contain foreground and background image difference information, but channel features within the same scale suffer from information isolation. Therefore, it's necessary to perform a separate three-layer fully connected (FC) processing on the difference features of each scale. The essence of FC is global interaction along the channel dimension, flattening the spatial dimension pixels of each scale feature map into channel dimension vectors, and using weight matrices to calculate linear combinations of different channel features.

[0038] Even after intra-scale enhancement, features at each scale still have limitations. For example, low-scale features are rich in detail but have low semantic abstraction, while high-scale features are semantically clear but have vague details. Therefore, multi-scale fusion from bottom to top (from high scale to low scale) is used to achieve complementary advantages of features at different scales, ultimately generating four sets of fused features (cat1-cat4). The multi-scale difference features are denoted from high to low resolution as the first scale difference feature, the second scale difference feature (DiffFeatureScale2), the third scale difference feature (DiffFeatureScale3), and the fourth scale difference feature (DiffFeatureScale4), respectively. The second scale difference feature (DiffFeatureScale2), the third scale difference feature (DiffFeatureScale3), and the fourth scale difference feature (DiffFeatureScale4) are then fused.

[0039] The specific steps for generating four sets of fused features (MFC) include: Generate the first fusion feature cat4: The original fourth-scale difference feature is processed twice with 4×4 convolution kernels and stride 2 using Dconv+BN+ReLU to generate the high-resolution first fusion feature cat4, which is used for accurate edge localization of large-volume foreign objects. The second fusion feature cat3 is generated by processing the original fourth-scale difference feature with a 4×4 convolution kernel and a stride of 2 using Dconv+BN+ReLU (which improves the resolution by 2 times), and then fusion it with the original third-scale difference feature using Add (semantic features are injected into detail features). After another deconvolution (4×4 convolution kernel and a stride of 2 using Dconv+BN+ReLU), a "semantic + mesoscale detail" fusion feature is generated. Generate the third fusion feature cat2: After performing a deconvolution operation on the original fourth-scale difference feature, it is fused with the original third-scale difference feature. After performing another deconvolution operation, it is fused with the original second-scale difference feature to generate a "semantic + small-scale detail" fusion feature, which is used for the localization of small-volume foreign objects. The fourth fusion feature cat1 is generated by performing two deconvolution operations on the original fourth-scale difference feature and one deconvolution operation on the original third-scale difference feature. These features are then fused with the original second-scale difference feature and then subjected to another convolution operation (to compress redundancy) to generate the full-scale collaborative fusion feature cat1, which is used to comprehensively determine the overall range of the foreign object.

[0040] Finally, by concat stacking the cat1-cat4 channels, multi-scale fusion encoding is completed, generating an integrated feature map that fully covers details and semantics.

[0041] The fused and encoded feature map is still an abstract channel feature, which needs to be decoded by the detection head to be transformed into a concrete spatial mask (foreign object mask). The core is feature compression, resolution restoration, category judgment, and size matching. The specific process is as follows: Convolution compression is performed through Conv1×1+BN+ReLU; resolution is improved through Dconv4×4+BN+ReLU; convolution refinement is performed through Conv3×3+BN+ReLU, and Conv1×1 convolution further compresses the channels to a single channel; sigmoid activation function judgment is performed, and the single channel feature value is mapped to the range [0,1] through the sigmoid function. A threshold (such as 0.5) is set. When it is greater than or equal to the set threshold, it is judged as a foreign object pixel, and when it is less than the set threshold, it is judged as a non-foreign object pixel; finally, through one upsampling, the feature map size is precisely adjusted to be consistent with the original input image to generate the foreign object mask and complete the foreign object detection.

[0042] Furthermore, multiple starting points are set for the target line, and a detection interval is formed between any two adjacent starting points. All starting points divide the line into multiple continuous detection intervals. The above-mentioned complete foreign object detection process for the track bed is executed for each detection interval to ensure that there are no blind spots in detection and to reduce the data processing complexity of a single interval, thereby improving the accuracy of identifying small foreign objects.

[0043] This embodiment provides a feature comparison-based method for detecting foreign objects in the ballast bed. This method not only improves detection accuracy by comparing multi-dimensional features of foreground and background images, reducing the false detection and missed detection rates of traditional threshold segmentation or manual inspection, but also achieves accurate identification of foreign objects of different sizes and types. It adapts to complex environments, solving interference problems such as changes in lighting, track stains, and irregularities in ballast particles, ensuring the robustness of the detection method in dynamic railway environments. Furthermore, it is highly automated and intelligent, reducing the frequency of manual visual inspections through automated detection algorithms, thus lowering the workload of railway maintenance personnel, especially in adverse weather conditions or nighttime operation scenarios.

[0044] In some embodiments, multi-scale feature extraction is performed on the registered foreground and background images to obtain the following extraction results: The registered foreground and background images are preprocessed to obtain feature maps, including convolution and pooling. The preprocessed feature maps are sequentially fed into feature extraction layers of different levels. Each feature extraction layer performs in-depth processing and downsampling of the features in the feature map through multiple bottleneck structures, generating multiple feature maps of different scales. The extraction result is obtained by packaging and integrating feature maps of different scales.

[0045] In some embodiments, each bottleneck structure includes three convolutional layers, batch normalization (BN), and a ReLU activation function. The three convolutional layers include 1×1 Conv dimensionality reduction, 3×3 Conv feature extraction, and 1×1 Conv dimensionality enhancement.

[0046] In some embodiments, extracting multi-scale difference features includes: Feature compression is performed on all scale feature maps after integration; Feature encoding is performed on the feature map after feature compression to extract the corresponding higher-order semantic information; The foreground features of the foreground image after feature encoding are used as the first dimension features, and the background features of the background image after feature encoding are used as the second dimension features. The foreground features and the background features are multiplied one by one according to the pixel position to obtain the third dimension features. The first dimension feature, the second dimension feature, and the third dimension feature are superimposed on the channel dimension to output multi-scale difference features.

[0047] In some embodiments, feature compression of all integrated scale feature maps includes: The number of feature channels in feature maps at different scales is compressed three times through convolutional layers; After the first compression, the number of feature channels becomes half the number of input channels plus the number of basic channels; After the second compression, the number of feature channels becomes four times the number of basic channels; After the third compression, the number of feature channels becomes twice the number of basic channels.

[0048] In some embodiments, detecting all multi-scale difference features and outputting foreign object detection results for the track bed includes: Each scale's difference features are processed separately using a three-layer fully connected layer; After three layers of fully connected processing, the scale difference features are fused and encoded across scales. Multi-scale difference features, after inter-scale fusion encoding, are decoded based on the detection head to output foreign objects in the track bed; The output of foreign objects in the track bed is taken as the detection result of foreign objects in the track bed.

[0049] In some embodiments, performing inter-scale fusion encoding includes: The multi-scale difference features corresponding to multiple feature maps at different scales are denoted from high to low resolution as the first scale difference feature, the second scale difference feature, the third scale difference feature, and the fourth scale difference feature, respectively. The fourth-scale difference features are subjected to two deconvolution operations to generate the first fusion feature; After performing a deconvolution operation on the fourth-scale difference feature, it is fused with the third-scale difference feature, and then after another deconvolution operation, a third fused feature is generated. After performing a deconvolution operation on the fourth-scale difference feature, it is fused with the third-scale difference feature. After another deconvolution operation, it is fused with the second-scale difference feature to generate the second fused feature. The fourth-scale difference feature is deconvolved twice, and the third-scale difference feature is deconvolved once. After fusing with the second-scale difference feature, a convolution operation is performed to generate the first fused feature. The first fusion feature, the second fusion feature, the third fusion feature, and the fourth fusion feature are superimposed on the channel dimension to complete the inter-scale fusion coding.

[0050] In some embodiments, the multi-scale difference features after inter-scale fusion encoding are decoded based on the detection head, including: The multi-scale difference features encoded by inter-scale fusion are sequentially processed by 1×1 convolution, deconvolution, 3×3 convolution, and Sigmoid activation function to complete feature decoding and upsampling, thereby obtaining the foreign objects in the track bed and completing the detection of foreign objects in the track bed.

[0051] In some embodiments, pixel-level registration of the depth map and grayscale image for the foreground starting image and all background starting images respectively includes: Obtain the grayscale image and depth image corresponding to the foreground starting image, and denot them as the first grayscale image and the first depth image, respectively. Obtain the grayscale and depth maps corresponding to all background starting images, and denote them as the second grayscale image and the second depth map, respectively. Feature points are extracted from the first grayscale image and all second grayscale images using a feature extraction algorithm. The similarity between the feature points in the first grayscale image and all second grayscale images is calculated using feature descriptors, and similar feature pairs are selected. Based on the first depth map and all second depth maps, extract the depth values ​​of each pair of feature points in the corresponding depth map, calculate the corresponding depth difference, and determine whether the depth difference is less than a preset threshold. If it is, it is determined to be a real feature pair; otherwise, it is determined to be a false feature pair. Based on the three-dimensional information of the first depth map and all second depth maps, extract the spatial coordinates of each pair of feature points in the real feature pair, and verify whether the spatial geometric relationship of each pair of feature points in the real feature pair is consistent. If yes, no action is taken; otherwise, the corresponding real feature pair is cleaned. Based on the cleaned real feature pairs, a spatial transformation model between the foreground and background images is constructed. Based on the spatial transformation model, the pixels of the background starting image are mapped to the coordinate system of the foreground starting image, thereby achieving pixel-level alignment between the foreground starting image and the background starting image.

[0052] In some embodiments, multiple starting points are set for the line, and a detection interval is formed between any adjacent starting points. All starting points divide the line into multiple detection intervals, and foreign object detection of the track bed is performed in each detection interval.

[0053] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computing software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0054] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0055] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for ballast foreign object detection based on feature contrast, characterized in that, The method comprises the following steps: Under the triggering of a position encoder, a track bed history image and corresponding position information in a track are collected, the track bed history image comprises a depth map and a grayscale map; The collected track bed history image and corresponding position information are bound as a background image and stored in a background database; A current track bed image and corresponding position information in the track are collected and bound as a foreground image, and a starting point image in the foreground image is extracted and recorded as a foreground starting image; Position information of the foreground starting image is obtained and recorded as foreground position information, a plurality of background starting images corresponding to the position of the foreground position information are searched in the background database, and all the background starting images are stored in a background starting image set; Pixel-level registration is performed on the foreground starting image and all images in the background starting image set to obtain a registered foreground image, a registered background image and a registration relationship, and subsequent foreground images and background images are registered based on the registration relationship; Multi-scale feature extraction is performed on the registered foreground image and the registered background image to obtain an extraction result; Based on the extraction result, multi-scale difference features are extracted by comparing the multi-scale features in the registered foreground image with the multi-scale features in the registered background image; All the multi-scale difference features are detected to output a track bed foreign matter detection result.

2. The method of claim 1, wherein, The multi-scale feature extraction on the registered foreground image and the registered background image to obtain the extraction result comprises: The registered foreground image and the registered background image are preprocessed to obtain a feature map, including convolution processing and pooling processing; The preprocessed feature map sequentially enters different levels of feature extraction layers, each feature extraction layer processes and down-samples features in the feature map through a plurality of bottleneck structures to generate a plurality of feature maps of different scales; The plurality of feature maps of different scales are packaged and integrated to obtain the extraction result.

3. The method of claim 2, wherein, Each bottleneck structure comprises three layers of convolution, BN batch normalization and ReLU activation function, and the three layers of convolution comprise 1x1Conv dimension reduction, 3x3Conv feature extraction and 1x1Conv dimension increase.

4. The method of claim 1, wherein, The multi-scale difference feature extraction comprises: Feature compression is performed on all the integrated scale feature maps; Feature encoding is performed on the feature maps after feature compression to extract corresponding high-order semantic information; A foreground feature of the foreground image after feature encoding is taken as a first dimension feature, a background feature of the background image after feature encoding is taken as a second dimension feature, and the foreground feature and the background feature are multiplied pixel by pixel as a third dimension feature; The first dimension feature, the second dimension feature and the third dimension feature are superimposed in the channel dimension to output the multi-scale difference feature.

5. The method of claim 4, wherein, The feature compression on all the integrated scale feature maps comprises: The feature channel number of the different scale feature maps is compressed three times through a convolution layer; After the first compression, the feature channel number becomes half of the input channel number plus a basic channel number; After the second compression, the feature channel number becomes 4 times of the basic channel number; After the third compression, the feature channel number becomes 2 times of the basic channel number.

6. The method of claim 1, wherein, The detection of all the multi-scale difference features to output the track bed foreign matter detection result comprises: The three-layer full connection processing is performed on each scale difference feature separately; Each scale difference feature after the three-layer full connection processing is subjected to inter-scale fusion encoding; The multi-scale difference feature after the inter-scale fusion encoding is decoded based on the detection head to output the ballast foreign matter; The output ballast foreign matter is taken as the ballast foreign matter detection result.

7. The method of claim 6, wherein, The inter-scale fusion encoding includes: The multi-scale difference features corresponding to the multiple different scale feature maps are denoted as second scale difference features, third scale difference features and fourth scale difference features from high to low in resolution; The fourth scale difference features are subjected to twice deconvolution operations to generate first fusion features; The fourth scale difference features are subjected to once deconvolution operation and fusion operation with the third scale difference features, and then subjected to once deconvolution operation to generate third fusion features; The fourth scale difference features are subjected to once deconvolution operation and fusion operation with the third scale difference features, and then subjected to once deconvolution operation and fusion operation with the second scale difference features to generate second fusion features; The fourth scale difference features are subjected to twice deconvolution operations and the third scale difference features are subjected to once deconvolution operation, and then subjected to fusion operation with the second scale difference features and once convolution operation to generate first fusion features; The first fusion features, the second fusion features, the third fusion features and the fourth fusion features are superimposed in the channel dimension to complete the inter-scale fusion encoding.

8. The method of claim 6, wherein the method further comprises: The multi-scale difference feature after the inter-scale fusion encoding based on the detection head decoding includes: The multi-scale difference feature after the inter-scale fusion encoding is subjected to 1×1 convolution operation, deconvolution operation, 3×3 convolution operation and Sigmoid activation function processing in sequence to complete feature decoding processing and once up-sampling to obtain the ballast foreign matter, and the detection of the ballast foreign matter is completed.

9. The method of claim 1, wherein, The pixel-level registration of the depth map and the grayscale map of the foreground starting image and all background starting images includes: The grayscale map and the depth map corresponding to the foreground starting image are obtained and denoted as a first grayscale map and a first depth map respectively; The grayscale map and the depth map corresponding to all background starting images are obtained and denoted as a second grayscale map and a second depth map respectively; Feature point extraction is performed on the first grayscale map and all second grayscale maps by using a feature extraction algorithm, and the similarity of the feature points in the first grayscale map and all second grayscale maps is calculated by using a feature descriptor to screen out similar feature pairs; Based on the first depth map and all second depth maps, the depth values of each pair of feature points in the corresponding depth maps in the similar feature pairs are extracted, and the corresponding depth difference values are calculated to determine whether the depth difference values are less than a preset threshold value, if yes, it is determined as a real feature pair, if not, it is determined as a false feature pair; Based on the three-dimensional information of the first depth map and all second depth maps, the spatial coordinates of each pair of feature points in the real feature pairs are extracted, and the spatial geometric relationship of each pair of feature points in the real feature pairs is verified, if yes, no action is taken, if not, the corresponding real feature pairs are cleaned. Based on the cleaned real feature pairs, a spatial transformation model of the foreground image and the background image is constructed; Based on the spatial transformation model, pixels of the background starting image are mapped to a coordinate system of the foreground starting image, realizing pixel-level alignment of the foreground starting image and the background starting image.

10. The method of claim 1-9, wherein, Also include: A plurality of starting points are arranged on the line, and a detection interval is formed between any adjacent starting points. All starting points divide the line into a plurality of detection intervals, and ballast foreign matter detection is performed on each detection interval.

Images