A multi-source data fusion track fastener defect detection method and system

Abstract

The application provides a track fastener defect detection method and system based on multi-source data fusion, which comprises the following steps: collecting a registered pair of fastener grayscale images and depth images; labeling the fastener defects in each pair of images and constructing a track fastener defect multi-modal dataset; inputting the fastener grayscale images and depth images in the dataset into a multi-modal track fastener defect detection model and training the model, detecting the output fastener defect detection result, and the model comprises a double-flow backbone network, a multi-modal feature fusion network and a head network; the double-flow backbone network is used for extracting the features of the input grayscale images and depth images in parallel; the multi-modal feature fusion network is used for fusing the features of different modalities and different scales; and the head network is used for identifying and positioning fastener defects of three different sizes, i.e., large, medium and small. The application can detect track fastener defects.

Description

Technical Field

[0001] This invention relates to the field of rail transit inspection technology, and in particular to a method and system for detecting defects in rail fasteners using multi-source data fusion. Background Technology

[0002] Track fasteners, mainly composed of components such as spiral spikes, gauge blocks, pads, elastic clips, and flat washers, are crucial infrastructure connecting rails and sleepers, serving to fix the rails and maintain track gauge. During long-term service, these fastener components may loosen or become missing due to factors such as train load, environmental corrosion, and material aging, posing a significant threat to train operation safety. Timely detection and handling of fastener defects are of paramount importance for ensuring safe train operation.

[0003] Traditional methods for detecting defects in fasteners primarily rely on manual inspection, which suffers from low efficiency, high labor intensity, and strong subjectivity, making it difficult to meet the real-time inspection needs of large-scale lines. Therefore, introducing automated inspection technology based on intelligent algorithms has become an inevitable trend in this field.

[0004] Most existing automated inspection technologies are based on two-dimensional image analysis, using image processing and deep learning algorithms to identify and locate defects. However, in engineering applications, the inherent limitations of two-dimensional images—such as susceptibility to light interference and lack of spatial structural information—lead to poor sensitivity of detection algorithms to small-sized defects and spatial deformation of fasteners, resulting in false alarms and missed detections. Summary of the Invention

[0005] To address the technical problems existing in the prior art, the present invention provides a method and system for detecting defects in track fasteners using multi-source data fusion, the technical solution of which is as follows: On the one hand, a method for detecting defects in track fasteners by multi-source data fusion is provided, the method comprising: S1. Use the image acquisition module installed at the rear of the track inspection trolley to scan the fasteners above the track and acquire grayscale and depth images of the registered fasteners. S2. Label the fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; S3. Input the grayscale image and depth image of the fastener in the dataset and train the multimodal track fastener defect detection model, and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small.

[0006] Optionally, the image acquisition module includes: a line structured light camera and a linear laser, wherein the laser line emitted by the linear laser is perpendicularly irradiated onto the track surface, and the laser cross-sectional image of the track is obtained by capturing the laser line through the line structured light camera; The image acquisition module acquires laser cross-sectional images of the track at equal intervals as the track inspection trolley moves, obtaining grayscale and depth information of the track. Finally, the acquired laser cross-sectional images are stitched together to generate a grayscale image of the fastener reflecting two-dimensional features and a depth image of the fastener reflecting three-dimensional features.

[0007] Optionally, the network structures of the gray-level branch and the depth branch of the dual-stream backbone network are each composed of the following modules connected in series: two nonlinear feature processing CBS modules, one attention and multi-branch convolution collaborative AMCS module, three composite structures, and one spatial pyramid pooling SPPF module.

[0008] Optionally, the CBS module is used to process the input grayscale image. and depth map Feature extraction and downsampling are performed to output a grayscale feature map. and depth feature map The system includes: a regular convolution module, a batch normalization module, and a SiLU activation function module; wherein, the regular convolution module is used for downsampling and changing the number of channels in the feature map; the batch normalization module is used for normalizing each channel of each batch of data to accelerate training; and the SiLU activation function module is used to introduce nonlinearity so that the neural network can fit complex nonlinear relationships. The AMCS module is used to extract long-range contextual information by combining attention mechanisms and multi-branch convolutional structures, thereby expanding the receptive field, enhancing the ability of the dual-stream backbone network to capture small-sized defect features, improving the model's detection performance for small-sized defects, and outputting enhanced grayscale feature maps. and enhanced deep feature maps ; The three composite structures are used for the... and Further feature extraction is performed separately, and grayscale feature maps are output sequentially. , , and depth feature map , , The three composite structures are all composed of a CBS module and a dual-branch residual C2f module connected in series. The C2f module is used for feature fusion and gradient optimization, and includes: a CBS module, a feature map separation module, N bottleneck modules connected in series, a channel stitching module, and a CBS module. The feature map separation module is used to divide the input feature map into multiple branches along the channel dimension to facilitate subsequent parallel processing. The bottleneck module is used to reduce the amount of computation and retain key features. The channel stitching module is used to stitch the feature map along the channel dimension. The SPPF module is used to achieve large-scale feature aggregation, enhance the network's ability to understand large targets and global structures, and output feature maps. and It includes: a regular convolutional module, three max pooling modules, a channel concatenation module, and another regular convolutional module; wherein, the regular convolutional module is used to downsample and change the number of channels in the feature map, the max pooling module is used to achieve large-scale feature aggregation, and the channel concatenation module is used to concatenate the feature map along the channel dimension.

[0009] Optionally, the AMCS module includes: a C2f module, a CBS module, a context anchor attention mechanism (CAA) module, and a multi-branch feature enhancement (FEM) module; Among them, for the input grayscale feature map The processing procedure is as follows: The The input is processed by the C2f module, and a feature map is output. , means as follows: At the same time, the above After processing by the CBS module, the data is input into the CAA module. Global contextual information of the image is extracted through average pooling, while long-strip depthwise convolution captures the correlation between each component of the fastener and its larger spatial neighborhood, compensating for the network's insufficient ability to extract contextual information. Spatial attention weights are then output. , means as follows: Then the and The product of and The outputs of the main branch are then weighted using spatial attention weights to guide the network to focus on the fastener and surrounding area, improving the recognition of small target parts and obtaining the first enhanced feature map. , means as follows: Then the The input is fed into the FEM module for feature enhancement, and the final enhanced feature map is output. , means as follows: .

[0010] Optionally, the CAA module, used to generate attention weights, includes: an average pooling module, a regular convolution module, two depthwise convolution modules, another regular convolution module, and a sigmoid activation function module; The average pooling module is used for downsampling and feature smoothing; the ordinary convolution module is used for downsampling and changing the number of channels in the feature map; the depthwise convolution module is used to reduce the number of parameters and computational cost through independent convolution of each channel; and the sigmoid activation function module is used to compress the input to the range of (0,1). The FEM module, used for feature enhancement, consists of four parallel branches. The outputs of the four branches are concatenated through a channel concatenation module, including: a residual branch for forming an equivalent mapping; a regular convolution branch for extracting local texture and edge features; and two dilated convolution branches for expanding the receptive field. The residual branch consists of a regular convolutional module, the regular convolutional branch consists of two regular convolutional modules connected in series, and the dilated convolutional branch consists of three regular convolutional modules and one dilated convolutional module connected in series. The regular convolutional module is used for downsampling and changing the number of channels in the feature map; the dilated convolutional module is used to capture multi-scale contextual information.

[0011] Optionally, the three A-GMFF modules employ a multi-stage dual-modal fusion and enhancement strategy. They receive shallow, mid-layer, and deep grayscale feature maps and depth feature maps from the dual-stream backbone network, respectively. This strategy aims to reduce feature differences while fully utilizing complementary information from different modalities, enhancing feature sharing and alignment, fully fusing features from two different modalities, and outputting three multi-modal fusion feature maps at three different scales. The shallow, mid-layer, and deep grayscale feature maps and depth feature maps from the dual-stream backbone network are respectively... , and The three different scales of multimodal fusion feature maps output are respectively , and ; The feature map is processed in the first A-GMFF module as follows: grayscale feature map and depth feature map The first fused feature map is obtained by concatenating the features along the channel dimension. , means as follows: The The input is fed into a regular convolutional module, and the output is a second fused feature map. , means as follows: The The input is fed into a convolutional pyramid structure consisting of three depthwise separable convolutional modules at different scales to extract mixed features, capture key features unique to each modality, learn the intrinsic relationships between different features to address the problem of insufficient semantic information in single-scale feature extraction, and output the mixed feature weights. , means as follows: grayscale feature map and depth feature map The inputs are respectively fed into a weight generation module to generate grayscale weights. and depth weight By explicitly modeling the interdependencies between channels, the channel feature responses are adaptively recalibrated to improve the quality of feature map representation. The weight generation module includes an adaptive average pooling module, a fully connected network module, and a sigmoid activation function module connected in series, as shown below: , The above and and The weights are then added together, and the depth and visible light weights are supplemented by the visible light and depth information from the hybrid feature weights, respectively, to generate the first enhancement weight. Second Enhancement Weight , means as follows: , Using the and For grayscale feature maps respectively and depth feature map Weighted processing is performed to generate enhanced grayscale feature maps. and enhanced deep feature maps , means as follows: , The and Add them together to generate the final fused feature map. , means as follows: .

[0012] Optionally, the feature pyramid structure, used to execute a top-down feature fusion path, includes: a first upsampling module, a first channel stitching module, a first C2f module, a second upsampling module, a second channel stitching module, and a second C2f module; The first upsampling module is used for... Upsampling is performed; the first channel stitching module is used to stitch the data along the channel dimension. and after upsampling The first C2f module is used to process the feature map processed by the first channel stitching module and output a first fused feature map. ; The second upsampling module is used for the... Upsampling is performed; the second channel stitching module is used to stitch the data along the channel dimension. and after upsampling The second C2f module is used to process the feature map processed by the second channel stitching module and output the second fused feature map. .

[0013] Optionally, the path aggregation network structure is used to execute a bottom-up feature enhancement strategy, including: a first CBS module, a third channel stitching module, a third C2f module, a second CBS module, a fourth channel stitching module, and a fourth C2f module; Wherein, the first CBS module is used for the... Feature extraction and downsampling are performed; the third channel stitching module is used to stitch the features along the channel dimension. and downsampling The third C2f module is used to process the feature map processed by the third channel stitching module and output the third fused feature map. ; The second CBS module is used for the... Feature extraction and downsampling are performed; the fourth channel stitching module is used to stitch the features along the channel dimension. and downsampling The fourth C2f module is used to process the feature map processed by the fourth channel stitching module and output the fourth fused feature map. .

[0014] On the other hand, a multi-source data fusion system for detecting defects in rail fasteners is provided, the system comprising: The acquisition module is used to scan the fasteners above the track using the image acquisition module installed at the rear of the track inspection trolley, and to acquire grayscale and depth images of the registered pairs of fasteners; The annotation construction module is used to annotate fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; The detection module is used to input the grayscale image and depth image of the fastener in the dataset and train a multimodal track fastener defect detection model, and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small.

[0015] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to realize the above-described method for detecting defects in track fasteners through multi-source data fusion.

[0016] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to realize the above-described method for detecting defects in track fasteners through multi-source data fusion.

[0017] The beneficial effects of the technical solution provided by this invention include at least the following: This invention provides a method and system for detecting defects in track fasteners using multi-source data fusion. A multimodal track fastener defect detection model is constructed for fastener defect detection. The model processes grayscale and depth maps in parallel through a dual-stream backbone network structure. In the two branches of the dual-stream backbone network, an attention and multi-branch convolutional collaborative AMCS module, a context anchor attention mechanism CAA module, and a multi-branch feature enhancement FEM module are introduced to expand the receptive field, strengthen the expression of local detailed features, and improve the model's detection performance for small-sized defects. The multimodal feature fusion network introduces three attention-guided dual-modal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure to fully explore the dual-modal feature fusion paradigm, generating more robust and more sensitive fusion features to defect morphology. Compared with commercially available single-modal track fastener defect detection models, the detection performance for fastener defects, especially small-sized defects such as missing flat washers and loose nuts, is significantly improved. This provides a reliable technical solution for intelligent inspection of railway systems and is of great significance for improving railway operation and maintenance efficiency and ensuring train operation safety. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for detecting defects in track fasteners using multi-source data fusion, provided by an embodiment of the present invention. Figure 2 This is a general block diagram of a multi-source data fusion method for detecting defects in rail fasteners provided in an embodiment of the present invention; Figure 3 This is a block diagram of the Attention and Multi-branch Convolution Collaboration (AMCS) module, the Context Anchor Attention (CAA) module, and the Multi-branch Feature Enhancement (FEM) module provided in this embodiment of the invention. Figure 4 This is a block diagram of the attention-guided multimodal feature fusion module A-GMFF provided in an embodiment of the present invention; Figure 5 This is a detection result diagram provided in an embodiment of the present invention; Figure 6 This is a block diagram of a multi-source data fusion-based track fastener defect detection system provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] To address the technical challenge that 2D images are susceptible to environmental interference and lack spatial structure information, resulting in poor sensitivity of 2D image-based detection methods to small-sized defects and spatial deformation of fasteners, leading to false positives and false negatives, this invention proposes a multi-source data fusion method for detecting track fastener defects. First, a dual-stream backbone network structure is constructed to extract grayscale features from grayscale images and depth features from depth images in parallel. Second, to address the poor detection performance of existing technologies for small-sized defects, this invention expands the receptive field and enhances local details by introducing an Attention and Multi-branch Convolution Collaboration (AMCS) module, a Context Anchor Attention (CAA) module, and a Multi-branch Feature Enhancement (FEM) module into the two branches of the dual-stream backbone network. Feature representation improves the model's detection performance for small-sized defects. Finally, due to the large differences in feature distribution between different modalities, existing feature layer fusion techniques tend to ignore the correlation between features of different modalities, and cannot effectively extract complementary information between different modalities, which can easily lead to information redundancy and affect model performance. In addition, compared with single-modal features, the fused bimodal features are more complex, and it is more difficult for the model to capture the features of small-sized fastener defects. The embodiments of this invention introduce three attention-guided bimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure into the multimodal feature fusion network to fully explore the bimodal feature fusion paradigm and generate fused features with higher robustness and greater sensitivity to defect morphology.

[0022] This invention provides a method for detecting defects in track fasteners by fusing multi-source data. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps:

[0023] S1. Use the image acquisition module installed at the rear of the track inspection trolley to scan the fasteners above the track and acquire grayscale and depth images of the registered fasteners. Optionally, the image acquisition module includes: a line structured light camera and a linear laser, wherein the laser line emitted by the linear laser is perpendicularly irradiated onto the track surface, and the laser cross-sectional image of the track is obtained by capturing the laser line through the line structured light camera; The image acquisition module acquires laser cross-sectional images of the track at equal intervals as the track inspection trolley moves, obtaining grayscale and depth information of the track. Finally, the acquired laser cross-sectional images are stitched together to generate a grayscale image of the fastener reflecting two-dimensional features and a depth image of the fastener reflecting three-dimensional features.

[0024] S2. Label the fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; In this embodiment of the invention, the labeling tool Labelimg is used to label fastener defects in each pair of images. The labeling categories include fastener, elastic strip miss, elastic strip loose, nut miss, nut loose, plain washer miss, plate miss, plate loose, baffle miss, baffle loose, foreign matter, component, and tool. A multimodal dataset of track fastener defects was constructed, and the image pairs in the dataset were divided into training set, validation set and test set in an 8:1:1 ratio.

[0025] S3. Input the grayscale image and depth image of the fastener in the dataset and train the multimodal track fastener defect detection model, and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small.

[0026] Optionally, such as Figure 2 As shown, the network structures of the gray-level branch and the depth branch of the dual-stream backbone network are composed of the following modules connected in series: two nonlinear feature processing CBS modules, one attention and multi-branch convolution collaborative AMCS module, three composite structures, and one spatial pyramid pooling SPPF module.

[0027] Optionally, the CBS module is used to process the input grayscale image. and depth map Feature extraction and downsampling are performed to output a grayscale feature map. and depth feature map The system includes: a regular convolution module, a batch normalization module, and a SiLU activation function module; wherein, the regular convolution module is used for downsampling and changing the number of channels in the feature map; the batch normalization module is used for normalizing each channel of each batch of data to accelerate training; and the SiLU activation function module is used to introduce nonlinearity so that the neural network can fit complex nonlinear relationships. The AMCS module is used to extract long-range contextual information by combining attention mechanisms and multi-branch convolutional structures, thereby expanding the receptive field, enhancing the ability of the dual-stream backbone network to capture small-sized defect features, improving the model's detection performance for small-sized defects, and outputting enhanced grayscale feature maps. and enhanced deep feature maps ; The three composite structures are used for the... and Further feature extraction is performed separately, and grayscale feature maps are output sequentially. , , and depth feature map , , The three composite structures are all composed of a CBS module and a dual-branch residual C2f module connected in series. The C2f module is used for feature fusion and gradient optimization, and includes: a CBS module, a feature map separation module, N bottleneck modules connected in series, a channel stitching module, and a CBS module. The feature map separation module is used to divide the input feature map into multiple branches along the channel dimension to facilitate subsequent parallel processing. The bottleneck module is used to reduce the amount of computation and retain key features. The channel stitching module is used to stitch the feature map along the channel dimension. In all embodiments of the present invention, the CBS module and C2f module have the same function.

[0028] The SPPF module is used to achieve large-scale feature aggregation, enhance the network's ability to understand large targets and global structures, and output feature maps. and It includes: a regular convolutional module, three max pooling modules, a channel concatenation module, and another regular convolutional module; wherein, the regular convolutional module is used to downsample and change the number of channels in the feature map, the max pooling module is used to achieve large-scale feature aggregation, and the channel concatenation module is used to concatenate the feature map along the channel dimension.

[0029] Optionally, such as Figure 3 As shown, the AMCS module includes: a C2f module, a CBS module, a context anchor attention mechanism (CAA) module, and a multi-branch feature enhancement (FEM) module; Among them, for the input grayscale feature map The processing procedure (for the input deep feature map) The processing procedure is similar, as follows: The The input is processed by the C2f module, and a feature map is output. , means as follows: At the same time, the above After processing by the CBS module, the data is input into the CAA module. Global contextual information of the image is extracted through average pooling, while long-strip depthwise convolution captures the correlation between each component of the fastener and its larger spatial neighborhood, compensating for the network's insufficient ability to extract contextual information. Spatial attention weights are then output. , means as follows: Then the and The product of and The outputs of the main branch are then weighted using spatial attention weights to guide the network to focus on the fastener and surrounding area, improving the recognition of small target parts and obtaining the first enhanced feature map. , means as follows: Then the The input is fed into the FEM module for feature enhancement, and the final enhanced feature map is output. , means as follows: .

[0030] Optionally, the CAA module, used to generate attention weights, includes: an average pooling module, a regular convolution module, two depthwise convolution modules, another regular convolution module, and a sigmoid activation function module; The average pooling module is used for downsampling and feature smoothing; the ordinary convolution module is used for downsampling and changing the number of channels in the feature map; the depthwise convolution module is used to reduce the number of parameters and computational cost through independent convolution of each channel; and the sigmoid activation function module is used to compress the input to the range of (0,1). The FEM module, used for feature enhancement, consists of four parallel branches. The outputs of the four branches are concatenated through a channel concatenation module, including: a residual branch for forming an equivalent mapping; a regular convolution branch for extracting local texture and edge features; and two dilated convolution branches for expanding the receptive field. The residual branch consists of a regular convolutional module, the regular convolutional branch consists of two regular convolutional modules connected in series, and the dilated convolutional branch consists of three regular convolutional modules and one dilated convolutional module connected in series. The regular convolutional module is used for downsampling and changing the number of channels in the feature map; the dilated convolutional module is used to capture multi-scale contextual information.

[0031] Optionally, such as Figure 4 As shown, the three A-GMFF modules employ a multi-stage dual-modal fusion and enhancement strategy. They receive shallow, mid-layer, and deep grayscale and depth feature maps from the dual-stream backbone network, respectively. This strategy aims to reduce feature differences while fully utilizing complementary information from different modalities, enhancing feature sharing and alignment, and fully fusing features from two different modalities. The resulting multi-modal fusion feature maps are output at three different scales. The shallow, mid-layer, and deep grayscale and depth feature maps from the dual-stream backbone network are respectively... , and The three different scales of multimodal fusion feature maps output are respectively , and ; The feature map processing procedure in the first A-GMFF module (the feature map processing procedure in other A-GMFF modules is similar) is as follows: grayscale feature map and depth feature map The first fused feature map is obtained by concatenating the features along the channel dimension. , means as follows: The The input is fed into a regular convolutional module, and the output is a second fused feature map. , means as follows: The The input is fed into a convolutional pyramid structure consisting of three depthwise separable convolutional modules at different scales to extract mixed features, capture key features unique to each modality, learn the intrinsic relationships between different features to address the problem of insufficient semantic information in single-scale feature extraction, and output the mixed feature weights. , means as follows: grayscale feature map and depth feature map The inputs are respectively fed into a weight generation module to generate grayscale weights. and depth weight By explicitly modeling the interdependencies between channels, the channel feature responses are adaptively recalibrated to improve the quality of feature map representation. The weight generation module includes an adaptive average pooling module, a fully connected network module, and a sigmoid activation function module connected in series, as shown below: , The above and and The weights are then added together, and the depth and visible light weights are supplemented by the visible light and depth information from the hybrid feature weights, respectively, to generate the first enhancement weight. Second Enhancement Weight , means as follows: , Using the and For grayscale feature maps respectively and depth feature map Weighted processing is performed to generate enhanced grayscale feature maps. and enhanced deep feature maps , means as follows: , The and Add them together to generate the final fused feature map. , means as follows: .

[0032] Optionally, the feature pyramid structure, used to execute a top-down feature fusion path, includes: a first upsampling module, a first channel stitching module, a first C2f module, a second upsampling module, a second channel stitching module, and a second C2f module; The first upsampling module is used for... Upsampling is performed; the first channel stitching module is used to stitch the data along the channel dimension. and after upsampling The first C2f module is used to process the feature map processed by the first channel stitching module and output a first fused feature map. ; The second upsampling module is used for the... Upsampling is performed; the second channel stitching module is used to stitch the data along the channel dimension. and after upsampling The second C2f module is used to process the feature map processed by the second channel stitching module and output the second fused feature map. .

[0033] Optionally, the path aggregation network structure is used to execute a bottom-up feature enhancement strategy, including: a first CBS module, a third channel stitching module, a third C2f module, a second CBS module, a fourth channel stitching module, and a fourth C2f module; Wherein, the first CBS module is used for the... Feature extraction and downsampling are performed; the third channel stitching module is used to stitch the features along the channel dimension. and downsampling The third C2f module is used to process the feature map processed by the third channel stitching module and output the third fused feature map. ; The second CBS module is used for the... Feature extraction and downsampling are performed; the fourth channel stitching module is used to stitch the features along the channel dimension. and downsampling The fourth C2f module is used to process the feature map processed by the fourth channel stitching module and output the fourth fused feature map. .

[0034] The head network includes three detection heads, each receiving... , and It is used to identify and locate fastener defects of three different sizes: large, medium, and small.

[0035] In this embodiment of the invention, the training set from the multimodal dataset of track fastener defects is input into the multimodal track fastener defect detection model for training. The training effect is verified using the validation set, resulting in a trained multimodal track fastener defect detection model. The trained multimodal track fastener defect detection model can be used to detect track fastener defects that need to be detected.

[0036] This invention uses Mean Average Precision (mAP) to evaluate model performance. mAP is the average of the Average Precision (AP) and is a key indicator for evaluating the overall performance of the model, where AP represents the average accuracy of the model in recognizing a specific target. Generally, mAP is expressed as mAP@0.5-0.95, representing the average mAP at each IoU value as the Intersection over Union (IoU) between the predicted and ground truth bounding boxes varies from 0.5 to 0.95 in steps of 0.05. The formula for calculating mAP is shown below:

[0037] Where P is precision, representing the proportion of correctly detected targets in the detection results; R is recall, representing the proportion of all targets successfully detected; and n represents the category of the detected samples. The operating system used for model training was Ubuntu 20.04, the GPU was NVIDIA GeForce RTX 4090, the training environment used Python version 3.8.20, PyTorch version 2.4.1, and CUDA version 11.3. During training, the image size was compressed to 640×640, the SGD optimizer was used, the batch size was set to 32, the initial learning rate was 0.001, and a total of 300 epochs were trained.

[0038] After training, the training effect was verified using a validation set, resulting in a multimodal track fastener defect detection model. The training results were then compared with commercially available single-modal track fastener defect detection models. The comparison results are shown in Table 1. Table 1 shows the mAP (0.5-0.95) of the improved model on the validation set for each sample. As shown in Table 1, compared with the single-modal YOLOv8n, the average mAP@0.5-0.95 of this model is improved by 4.7%, with mAP@0.5-0.95 for the three smallest defects—missing nuts, loose nuts, and missing flat washers—improving by 8.8%, 9.3%, and 13.7%, respectively. Furthermore, compared with other models, this model achieves the best performance across all 13 sample categories, demonstrating its superiority.

[0039] The test set is input into the trained multimodal track fastener defect detection model for detection, and the detection results are obtained. Figure 5 This is the detection result image of this model, from Figure 5As can be seen from the results, this model detected all defects in the five sets of test images. It also maintained a high confidence level when facing three types of small defects: loose nuts, missing flat washers, etc., demonstrating the powerful performance of the multimodal track fastener defect detection model.

[0040] like Figure 6 As shown in the figure, this embodiment of the invention also provides a multi-source data fusion-based track fastener defect detection system, the system comprising: The acquisition module 610 is used to scan fasteners above the track using an image acquisition module installed at the rear of the track inspection trolley, and to acquire grayscale and depth images of the registered pairs of fasteners. The annotation construction module 620 is used to annotate fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; The detection module 630 is used to input the grayscale image and depth image of the fastener in the dataset and train a multimodal track fastener defect detection model, and detect and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small.

[0041] The multi-source data fusion track fastener defect detection system provided in this embodiment of the invention has a functional structure that corresponds to the multi-source data fusion track fastener defect detection method provided in this embodiment of the invention, and will not be described again here.

[0042] Figure 7 This is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present invention. The electronic device 700 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 701 and one or more memories 702. The memory 702 stores at least one instruction, which is loaded and executed by the processor 701 to implement the steps of the above-mentioned multi-source data fusion method for detecting defects in track fasteners.

[0043] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the above-described method for detecting defects in track fasteners through multi-source data fusion. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0044] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting defects in rail fasteners using multi-source data fusion, characterized in that, The method includes: S1. Use the image acquisition module installed at the rear of the track inspection trolley to scan the fasteners above the track and acquire grayscale and depth images of the registered fasteners. S2. Label the fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; S3. Input the grayscale image and depth image of the fastener in the dataset and train the multimodal track fastener defect detection model, and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small. The network structures of the gray-level branch and the depth branch of the dual-stream backbone network are all composed of the following modules connected in series: two non-linear feature processing CBS modules, one attention and multi-branch convolution collaborative AMCS module, three composite structures, and one spatial pyramid pooling SPPF module. The three A-GMFF modules employ a multi-stage dual-modal fusion and enhancement strategy. They receive shallow, mid-layer, and deep grayscale and depth feature maps from the dual-stream backbone network, respectively. This strategy aims to reduce feature differences while fully utilizing complementary information from different modalities, enhancing feature sharing and alignment, and fully fusing features from two different modalities. The resulting multi-modal fusion feature maps are output at three different scales. The shallow, mid-layer, and deep grayscale and depth feature maps from the dual-stream backbone network are respectively... , and The three different scales of multimodal fusion feature maps output are respectively , and ; The feature map is processed in the first A-GMFF module as follows: grayscale feature map and depth feature map The first fused feature map is obtained by concatenating the features along the channel dimension. , means as follows: The The input is fed into a regular convolutional module, and the output is a second fused feature map. , means as follows: The The input is fed into a convolutional pyramid structure consisting of three depthwise separable convolutional modules at different scales to extract mixed features, capture key features unique to each modality, learn the intrinsic relationships between different features to address the problem of insufficient semantic information in single-scale feature extraction, and output the mixed feature weights. , means as follows: grayscale feature map and depth feature map The inputs are respectively fed into a weight generation module to generate grayscale weights. and depth weight By explicitly modeling the interdependencies between channels, the channel feature responses are adaptively recalibrated to improve the quality of feature map representation. The weight generation module includes an adaptive average pooling module, a fully connected network module, and a sigmoid activation function module connected in series, as shown below: ， The above and and The weights are then added together, and the depth and visible light weights are supplemented by the visible light and depth information from the hybrid feature weights, respectively, to generate the first enhancement weight. Second Enhancement Weight , means as follows: ， Using the and For grayscale feature maps respectively and depth feature map Weighted processing is performed to generate enhanced grayscale feature maps. and enhanced deep feature maps , means as follows: , The and The features are added together to generate the final fused feature map. , means as follows: 。 2. The method according to claim 1, characterized in that, The image acquisition module includes a line structured light camera and a linear laser. The laser line emitted by the linear laser is perpendicularly irradiated onto the track surface, and the laser cross-sectional image of the track is obtained by capturing the laser line through the line structured light camera. The image acquisition module acquires laser cross-sectional images of the track at equal intervals as the track inspection trolley moves, obtaining grayscale and depth information of the track. Finally, the acquired laser cross-sectional images are stitched together to generate a grayscale image of the fastener reflecting two-dimensional features and a depth image of the fastener reflecting three-dimensional features.

3. The method according to claim 1, characterized in that, The CBS module is used to process the input grayscale image. and depth map Feature extraction and downsampling are performed to output a grayscale feature map. and depth feature map The system includes: a regular convolution module, a batch normalization module, and a SiLU activation function module; wherein, the regular convolution module is used for downsampling and changing the number of channels in the feature map; the batch normalization module is used for normalizing each channel of each batch of data to accelerate training; and the SiLU activation function module is used to introduce nonlinearity so that the neural network can fit complex nonlinear relationships. The AMCS module is used to extract long-range contextual information by combining attention mechanisms and multi-branch convolutional structures, thereby expanding the receptive field, enhancing the ability of the dual-stream backbone network to capture small-sized defect features, improving the model's detection performance for small-sized defects, and outputting enhanced grayscale feature maps. and enhanced deep feature maps ; The three composite structures are used for the... and Further feature extraction is performed separately, and grayscale feature maps are output sequentially. , , and depth feature map , , The three composite structures are all composed of a CBS module and a dual-branch residual C2f module connected in series. The C2f module is used for feature fusion and gradient optimization, and includes: a CBS module, a feature map separation module, N bottleneck modules connected in series, a channel stitching module, and a CBS module. The feature map separation module is used to divide the input feature map into multiple branches along the channel dimension to facilitate subsequent parallel processing. The bottleneck module is used to reduce the amount of computation and retain key features. The channel stitching module is used to stitch the feature map along the channel dimension. The SPPF module is used to achieve large-scale feature aggregation, enhance the network's ability to understand large targets and global structures, and output feature maps. and It includes: a regular convolutional module, three max pooling modules, a channel concatenation module, and another regular convolutional module; wherein, the regular convolutional module is used to downsample and change the number of channels in the feature map, the max pooling module is used to achieve large-scale feature aggregation, and the channel concatenation module is used to concatenate the feature map along the channel dimension.

4. The method according to claim 3, characterized in that, The AMCS module includes: a C2f module, a CBS module, a context anchor attention mechanism (CAA) module, and a multi-branch feature enhancement (FEM) module; Among them, for the input grayscale feature map The processing procedure is as follows: The The input is processed by the C2f module, and a feature map is output. , means as follows: At the same time, the above After processing by the CBS module, the data is input into the CAA module. Global contextual information of the image is extracted through average pooling, while long-strip depthwise convolution captures the correlation between each component of the fastener and its larger spatial neighborhood, compensating for the network's insufficient ability to extract contextual information. Spatial attention weights are then output. , means as follows: Then the and The product of and The outputs of the main branch are then weighted using spatial attention weights to guide the network to focus on the fastener and its surrounding area, improving the recognition of small target parts and resulting in the first enhanced feature map. , means as follows: Then the The input is fed into the FEM module for feature enhancement, and the final enhanced feature map is output. , means as follows: 。 5. The method according to claim 4, characterized in that, The CAA module, used to generate attention weights, includes: an average pooling module, a regular convolution module, two depthwise convolution modules, another regular convolution module, and a sigmoid activation function module. The average pooling module is used for downsampling and feature smoothing; the ordinary convolution module is used for downsampling and changing the number of channels in the feature map; the depthwise convolution module is used to reduce the number of parameters and computational cost through independent convolution of each channel; and the sigmoid activation function module is used to compress the input to the range of (0,1). The FEM module, used for feature enhancement, consists of four parallel branches. The outputs of the four branches are concatenated through a channel concatenation module, including: a residual branch for forming an equivalent mapping; a regular convolution branch for extracting local texture and edge features; and two dilated convolution branches for expanding the receptive field. The residual branch consists of a regular convolutional module, the regular convolutional branch consists of two regular convolutional modules connected in series, and the dilated convolutional branch consists of three regular convolutional modules and one dilated convolutional module connected in series. The regular convolutional module is used for downsampling and changing the number of channels in the feature map; the dilated convolutional module is used to capture multi-scale contextual information.

6. The method according to claim 1, characterized in that, The feature pyramid structure is used to execute a top-down feature fusion path, including: a first upsampling module, a first channel stitching module, a first C2f module, a second upsampling module, a second channel stitching module, and a second C2f module; The first upsampling module is used for... Upsampling is performed; the first channel stitching module is used to stitch the data along the channel dimension. and after upsampling The first C2f module is used to process the feature map processed by the first channel stitching module and output a first fused feature map. ; The second upsampling module is used for the... Upsampling is performed; the second channel stitching module is used to stitch the data along the channel dimension. and after upsampling The second C2f module is used to process the feature map processed by the second channel stitching module and output the second fused feature map. .

7. The method according to claim 6, characterized in that, The path aggregation network structure is used to execute a bottom-up feature enhancement strategy, including: a first CBS module, a third channel splicing module, a third C2f module, a second CBS module, a fourth channel splicing module, and a fourth C2f module; Wherein, the first CBS module is used for the... Feature extraction and downsampling are performed; the third channel stitching module is used to stitch the features along the channel dimension. and downsampling The third C2f module is used to process the feature map processed by the third channel stitching module and output the third fused feature map. ; The second CBS module is used for the... Feature extraction and downsampling are performed; the fourth channel stitching module is used to stitch the features along the channel dimension. and downsampling The fourth C2f module is used to process the feature map processed by the fourth channel stitching module and output the fourth fused feature map. .

8. A multi-source data fusion system for detecting defects in rail fasteners, characterized in that, The system for implementing the multi-source data fusion method for detecting defects in track fasteners according to any one of claims 1-7, the system comprising: The acquisition module is used to scan the fasteners above the track using the image acquisition module installed at the rear of the track inspection trolley, and to acquire grayscale and depth images of the registered pairs of fasteners; The annotation construction module is used to annotate fastener defects in each pair of images and construct a multimodal dataset of track fastener defects; The detection module is used to input the grayscale image and depth image of the fastener in the dataset and train a multimodal track fastener defect detection model, and output the fastener defect detection result. The multimodal track fastener defect detection model includes: a dual-stream backbone network, a multimodal feature fusion network and a head network. The dual-stream backbone network is used to extract features of the input grayscale image and depth image in parallel, including: grayscale branches and depth branches set in parallel and having the same network structure; The multimodal feature fusion network is used to fuse features of different modalities and scales, and includes: three attention-guided multimodal feature fusion A-GMFF modules, a feature pyramid structure, and a path aggregation network structure; The head network includes three detection heads for identifying and locating fastener defects of three different sizes: large, medium, and small.

Images