A rail defect detection method based on frequency information and cross-modal fusion strategy

Abstract

The present application relates to the technical field of computer vision, and more particularly to a rail defect detection method based on frequency information and cross-modal fusion strategy, comprising the following steps: S1, pre-processing the RGB-D rail image; S2, inputting the RGB image generated by S1 into a first encoder based on SMT network construction, and inputting the depth image generated by S1 into a second encoder based on MobileNetV2 network construction, and extracting five levels of RGB feature maps and depth feature maps respectively; S3, frequency domain enhancement and low layer fusion: for the RGB feature maps and depth feature maps of Stage1 and Stage2 levels in S2, sequentially execute: 3a frequency domain dynamic enhancement; 3b low layer cross-modal fusion; S4, high layer feature cross-modal fusion; S5, cross-scale feature fusion; S6, defect detection: output the position and range of rail surface defects. This method can significantly improve the detection rate of small defects and anti-interference ability while ensuring efficient deployment of the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a rail defect detection method based on frequency information and a cross-modal fusion strategy. Background Technology

[0002] Railway transportation, as the main artery of the modern logistics and transportation system, bears the heavy responsibility of the nation's economic lifeline. As the core infrastructure of the railway system, the structural integrity and surface quality of rails directly affect the safety, stability, and operational efficiency of train operation. However, manufacturing errors are unavoidable, and coupled with the high-frequency, high-load train rolling during long-term service, rail surfaces are highly susceptible to various defects, such as cracks, holes, peeling, and abrasions. If these defects are not identified and addressed in a timely manner, they can accelerate track deterioration or even lead to major safety accidents such as derailments, causing severe economic losses and social impact. Therefore, developing an efficient, accurate, and robust automatic detection method for rail surface defects is of great significance for ensuring railway transportation safety.

[0003] Currently, most mainstream rail defect detection methods rely on single-modal visual information (such as using only RGB images or depth maps), extract features through traditional convolutional neural networks, and then combine them with simple cross-scale fusion strategies for defect localization. These methods face two major bottlenecks in practical applications:

[0004] First, existing models typically employ a single or structurally redundant encoder architecture, lacking collaborative optimization designs tailored to the characteristics of multimodal data. For example, the absence of a dual-stream lightweight encoder based on Swing Transformer (SMT) and MobileNetV2 makes it difficult to balance global semantic modeling capabilities and computational efficiency during feature extraction, resulting in large model size, slow inference speed, and difficulty in deployment on edge devices. Simultaneously, the feature fusion process lacks hierarchy and specificity, failing to effectively integrate the rich texture information of RGB images with the geometric structure information of depth images, thus limiting further improvements in detection accuracy.

[0005] Secondly, existing methods generally neglect the utilization of video domain information and lack explicit modeling of image frequency components. Rail surfaces often exhibit texture interference highly similar to real defects due to metallic reflection, oil stains, or oxidation, and traditional spatial domain methods struggle to effectively distinguish these "pseudo-defects." Introducing discrete wavelet transform (such as Haar wavelets) for frequency domain decomposition of multimodal features, combined with a gated dynamic fusion mechanism to adaptively enhance low-frequency (structural) and high-frequency (detail) components, would significantly improve the model's sensitivity to weak signal defects such as small cracks and micropores. However, the effective coupling of frequency and spatial domain features, the dynamic weighted fusion of multimodal frequency domain information, and the balance between high-frequency detail preservation and low-frequency semantic enhancement while maintaining lightweight design involve complex signal processing and attention mechanism co-design, making technical implementation quite challenging.

[0006] Therefore, how to significantly improve the detection rate and anti-interference ability of small defects while ensuring efficient model deployment has become an urgent problem to be solved. Summary of the Invention

[0007] In view of the above-mentioned shortcomings of the existing technology, the purpose of this invention is to provide a rail defect detection method based on frequency information and cross-modal fusion strategy, which can significantly improve the detection rate and anti-interference ability of small defects while ensuring efficient model deployment.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A rail defect detection method based on frequency information and cross-modal fusion strategy includes the following steps:

[0010] S1. Preprocess the RGB-D rail image to generate a uniformly sized RGB image, a normalized depth image, and corresponding pixel-level binary mask labels.

[0011] S2. Input the RGB image generated in S1 into the first encoder built on the SMT network, and input the depth image generated in S1 into the second encoder built on the MobileNetV2 network to extract RGB feature maps and depth feature maps at five levels respectively.

[0012] S3, Frequency Domain Enhancement and Low-Level Fusion: For the RGB feature maps and depth feature maps of Stage 1 and Stage 2 in S2, perform the following sequentially:

[0013] 3a. Frequency Domain Dynamic Enhancement: Processing is performed through the frequency domain dynamic enhancement module FDEM. The FDEM module first generates a fused feature map based on the RGB feature map and the depth feature map, and then performs discrete wavelet transform on the fused feature map to extract low-frequency and high-frequency components. Subsequently, using the low-frequency and high-frequency components, the original RGB feature map and the depth feature map are weighted and enhanced respectively through a gated dynamic fusion mechanism, and the enhanced RGB feature map and the enhanced depth feature map are output.

[0014] 3b Low-level cross-modal fusion: The enhanced RGB feature map output from 3a and the depth feature map are input into the Interaction Unit (IIU) module for fusion. The IIU module first performs feature concatenation and enhancement on the two, then extracts spatial location-aware features based on the coordinate attention mechanism, and combines multi-scale spatial context information. Finally, it performs global weighted fusion through the spatial attention mechanism to output low-level fused features.

[0015] S4. High-level feature cross-modal fusion: For the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2, the cross-interaction fusion module CIFM is used for fusion. The CIFM module first enhances the details of the bimodal features through the detail-aware unit DPU, and then extracts spatial position-aware features and channel context features based on spatial attention mechanism and coordinate attention mechanism respectively. The two are then integrated collaboratively through multi-path interaction fusion strategy to output high-level bimodal features.

[0016] S5. Cross-scale feature fusion: Construct a multi-scale feature representation composed of the low-level fusion features output by S3 and the high-level bimodal features output by S4. Perform top-down layer-by-layer fusion through the cross-scale fusion module CF. At each fusion layer, the CF module upsamples the fusion features of the previous level, generates multi-dimensional guiding weights based on channel attention and foreground perception mechanisms, and uses the multi-dimensional guiding weights to perform weighted fusion of the multi-scale features of the current level to obtain enhanced features that fuse high-level semantics and low-level details. The enhanced features of the lowest level are used as the final fusion features.

[0017] S6. Defect Detection: The final fused features output from S5 are decoded to generate a defect prediction probability map. The probability map is then thresholded to output the location and range of defects on the rail surface.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. Dual encoder collaborative design balances performance and efficiency. It employs an RGB encoder based on Swing Transformer (SMT) and a lightweight MobileNetV2 depth encoder, respectively adapting to the needs of rich textures and geometric sensitivity. Compared to traditional single-network or symmetric dual-stream structures, this heterogeneous encoding strategy effectively controls model complexity while retaining strong expressive power, making it more suitable for resource-constrained deployment scenarios.

[0020] 2. Frequency Domain Dynamic Enhancement Improves the Ability to Detect Weak Defects. Discrete wavelet transform is introduced at the low-level feature stage to explicitly separate low-frequency (overall structure) and high-frequency (edge ​​details) components, and the original multimodal features are dynamically modulated through a gating mechanism. Compared to existing methods that rely solely on spatial domain convolution, this design effectively amplifies the high-frequency response of the defect region while suppressing false positives caused by interference factors such as metal reflection and scratches, significantly enhancing the ability to identify weak signal defects such as fine cracks and shallow pits.

[0021] 3. Layered Differentiated Fusion Strategy Optimizes Feature Complementarity. For different layers of network features, low-level Interaction Units (IIUs) and high-level Cross-Interaction Fusion Modules (CIFM) are designed: low-level fusion emphasizes spatial location awareness and multi-scale context integration, preserving fine geometric details; high-level fusion, through detail-aware units and a dual attention mechanism, strengthens semantic consistency and critical channel response. This layered, customized fusion approach overcomes the semantic ambiguity or detail loss problems caused by traditional unified fusion strategies.

[0022] 4. Cross-scale guided fusion achieves precise semantic-detail alignment. A top-down cross-scale fusion module (CF) is used, combining channel attention and foreground awareness mechanisms to generate multi-dimensional guided weights, dynamically adjusting the contribution of features at each level. Compared to conventional upsampling followed by direct concatenation or addition, this mechanism more effectively and accurately transmits high-level semantic information to the output layer, generating defect prediction results with clear boundaries and accurate localization.

[0023] In summary, this method can significantly improve the detection rate and anti-interference ability of small defects while ensuring efficient model deployment.

[0024] Preferably, steps S2 to S5 are performed by a detection network, which includes a dual-stream encoder, an FDEM module, an IIU module, a CIFM module, and a CF module, and is trained through end-to-end supervised training. The training uses an RGB-D rail image dataset with pixel-level binary mask labels, and optimizes the network parameters using a joint loss function consisting of binary cross-entropy loss and cross-union ratio loss.

[0025] This technical solution, through the construction of an end-to-end trainable unified detection network, organically integrates a dual-stream encoder with multi-level fusion modules (FDEM, IIU, CIFM, CF), and employs pixel-level label supervision and joint loss functions for optimization. This effectively achieves collaborative learning of multimodal feature extraction, frequency domain enhancement, and cross-scale fusion. Compared to traditional methods that rely on staged training or manually designed fusion rules, this integrated architecture avoids information loss and error accumulation in intermediate stages, significantly improving the overall convergence stability and defect localization accuracy of the model.

[0026] Preferably, in S1, the preprocessing includes: uniformly adjusting the resolution of the RGB image and the depth image to 256×256 pixels, and normalizing the depth image.

[0027] This setup, by uniformly adjusting the resolution of RGB and depth images to 256×256 pixels and normalizing the depth map, effectively eliminates the heterogeneity of multimodal data in terms of spatial scale and numerical distribution. This preprocessing strategy not only ensures the consistency of the input dimensions of the dual-stream encoder, avoiding additional interpolation errors or alignment computational overhead caused by size mismatch, but also maps depth information to a stable numerical range, significantly accelerating the convergence process of network gradients and improving training stability.

[0028] Preferably, in S2, the five extracted hierarchical feature maps correspond to Stage 1 to Stage 5, and their spatial resolutions are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original input image size, respectively; both the SMT network and the MobileNetV2 network are initialized using model parameters pre-trained on the ImageNet dataset.

[0029] This setup initializes the model using SMT and MobileNetV2 models pre-trained on ImageNet and constructs a multi-scale feature pyramid ranging from 1 / 4 to 1 / 64, effectively utilizing general texture and geometric prior knowledge learned from large-scale natural image data. This strategy significantly mitigates the risk of model overfitting caused by the scarcity of industrial defect samples and accelerates network convergence. Simultaneously, the multi-level resolution feature representation ensures that the network can capture global semantic context through deep, low-resolution features while preserving the fine spatial structure of minute defects using shallow, high-resolution features, overcoming the limitation of single-scale features in simultaneously achieving both localization accuracy and classification accuracy. This design significantly improves the model's generalization performance and multi-scale defect adaptability under small sample conditions, providing a solid feature foundation for accurate detection in complex backgrounds.

[0030] Preferably, in step 3a of S3, the FDEM module includes a fused feature map generation unit, a frequency domain decomposition unit, and a frequency domain fusion unit (FFU), used to dynamically enhance the RGB feature map and the depth feature map in the frequency domain, as follows:

[0031] First, the upsampled RGB feature map and the upsampled depth feature map are concatenated by the fusion feature map generation unit, and a preliminary fusion feature map is obtained by 3×3 convolution.

[0032] Then, through the frequency domain decomposition unit, a discrete wavelet transform is performed on the preliminary fused feature map to extract its low-frequency and high-frequency components, and directional detail extraction is performed on the high-frequency components to obtain high-frequency information in the horizontal, vertical and diagonal directions, and then they are stitched together to form a multi-directional high-frequency feature map.

[0033] Finally, the low-frequency component and the multi-directional high-frequency feature map are dynamically fused with the original RGB feature map and depth feature map through the frequency domain fusion unit (FFU). The dynamic fusion includes: controlling the contribution ratio of low-frequency and high-frequency features using learnable weight parameters, and adding the weighted frequency domain features to the original feature map after spatial adjustment by 3×3 convolution to obtain the enhanced RGB feature map and depth feature map.

[0034] This setup offers several advantages: 1. Enhanced response to subtle defects through explicit multi-directional frequency domain detail modeling. By using discrete wavelet transform to decompose features into low-frequency structures and high-frequency components in multiple directions (horizontal, vertical, and diagonal), this design overcomes the limitation of traditional spatial convolution in explicitly capturing textures in specific directions. For subtle defects with clear directions, such as cracks and scratches on rail surfaces, the extraction and utilization of multi-directional high-frequency information can significantly amplify their edge response. Simultaneously, the low-frequency components maintain the stability of the background structure, effectively solving the problem of subtle defect features being easily obscured under complex lighting conditions.

[0035] 2. Gated dynamic fusion mechanism achieves adaptive feature modulation. Learnable weight parameters are introduced to control the contribution ratio of frequency domain components to the original features, enabling the network to dynamically adjust the enhancement strategy based on the characteristics of the input content. Compared to feature addition or concatenation with fixed weights, this gating mechanism intelligently suppresses high-frequency noise interference such as metallic reflections and oil stains, enhancing high-frequency details only when identified as defective areas. This improves the signal-to-noise ratio while avoiding artifacts caused by over-enhancement, achieving both accuracy and adaptability in feature enhancement.

[0036] This module significantly improves the model's sensitivity to directional micro-defects and its noise robustness through reconstruction and dynamic modulation from a frequency domain perspective, providing high-purity feature inputs for subsequent cross-modal fusion.

[0037] Preferably, in step 3b of S3, the IIU module includes a stitching unit, a CBR structure, a coordinate attention mechanism, a depthwise separable convolution, a multi-scale convolution branch, and a spatial attention mechanism, used to interact and fuse the enhanced RGB feature map and the enhanced depth feature map output in step 3a, as follows:

[0038] First, the enhanced RGB feature map and the enhanced depth feature map are concatenated to obtain the enhanced fused feature map;

[0039] Then, the concatenated enhanced fused feature map is subjected to feature enhancement processing through a CBR structure to improve the feature representation capability; the CBR structure includes convolutional layers, batch normalization, and ReLU activation function;

[0040] Next, spatial attention weights are generated through a coordinate attention mechanism and applied to the enhanced fusion feature map processed by the CBR structure to generate a spatially aware feature map.

[0041] Then, a 5×5 depthwise separable convolution is performed on the spatially perceived feature map to obtain a spatial relationship enhancement feature map;

[0042] Then, spatial correlation information at different scales is extracted by three convolutional branches of different scales, and the output of each branch is added element by element to the spatial relationship enhancement feature map to obtain a multi-scale fusion feature map.

[0043] Finally, a spatial attention mechanism is used to perform global spatial modeling on the multi-scale fused feature map, and low-level fused features are output.

[0044] This setup achieves precise spatial localization and lightweight computation through the synergy of coordinate attention and depthwise separable convolutions. The introduction of coordinate attention embeds positional information into channel attention, overcoming the limitation of traditional global pooling in losing spatial coordinates, enabling the network to accurately pinpoint the specific location of rail defects. Combined with 5×5 depthwise separable convolutions, it effectively expands the receptive field to capture local spatial context while significantly reducing the number of parameters and computational complexity. This combination ensures that the model maintains real-time performance while possessing a strong ability to perceive and locate the spatial structure of minute defects.

[0045] 2. Multi-scale convolutional branches and global spatial attention construct multi-level feature associations. By setting three convolutional branches of different scales in parallel, the network can simultaneously capture multi-granular spatial association information ranging from fine textures to macroscopic morphology, solving the problem that single-scale convolution cannot adapt to the drastic changes in the size of rail defects. Subsequently, a spatial attention mechanism is used for global modeling to further suppress background noise and highlight key defect areas. This "multi-scale extraction + global optimization" strategy achieves deep fusion of RGB texture information and depth geometric information in the spatial dimension, significantly improving the robustness of defect detection in complex backgrounds.

[0046] This module enhances the interaction efficiency of multimodal features through refined spatial attention guidance and efficient multi-scale feature aggregation. It also constructs a feature representation system that balances positioning accuracy, scale adaptability, and computational efficiency, providing high-quality low-level fusion features for subsequent high-level semantic reasoning.

[0047] Preferably, in S4, the CIFM module includes a detail-aware unit (DPU), a spatial attention mechanism (SA), a coordinate attention mechanism (CA), and a feature fusion unit, used to perform layer-by-layer high-level cross-modal fusion of RGB feature maps and depth feature maps from Stage 3 to Stage 5 in S2, as follows:

[0048] First, the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2 are fused and enhanced with details by the detail-aware unit (DPU) to obtain the enhanced joint features of the corresponding levels.

[0049] Then, for each level, the enhanced joint features of that level are input into the spatial attention mechanism SA to generate spatial weights, which are then applied to the enhanced joint features of that level to obtain spatial enhanced features.

[0050] Next, for each level, the spatial augmentation features of that level are input into the coordinate attention mechanism (CA) to generate channel weights, which are then applied to the spatial augmentation features of that level to obtain channel augmentation features.

[0051] Then, for each level, the spatial enhancement features and channel enhancement features of that level are added and multiplied to obtain the first fusion feature and the second fusion feature, respectively.

[0052] Finally, the first and second fusion features of all levels are concatenated separately, and the channel dimensions are adjusted by 1×1 convolution to output the final high-level bimodal features.

[0053] This setup achieves precise focusing on defect regions through a serial "spatial-channel" dual attention mechanism. The scheme innovatively employs a serial processing strategy of first spatial attention (SA) and then coordinate attention (CA), forming a hierarchical filtering mechanism from "where is important" to "which features are important." SA first suppresses complex background noise and locks the spatial location of the defect, while CA then uses coordinate information to enhance the response of specific channels to the texture and geometry of the defect. This coarse-to-fine dual filtering effectively solves the problem of rich semantic information but blurred spatial details in high-level features, significantly improving the model's ability to discriminate small defects in strong interference backgrounds.

[0054] 2. The dual-path fusion strategy of addition and multiplication balances feature complementarity and nonlinear interaction. A dual-path feature aggregation is constructed by element-wise addition and multiplication of spatial and channel-enhanced features, respectively. The addition operation preserves the linear integrity of the original features, preventing the loss of key information; the multiplication operation introduces a strong nonlinear gating effect, forcing the network to focus on high-confidence regions where both features co-respond. This parallel fusion approach of "preservation + enhancement" maximizes the complementary advantages of RGB texture information and depth geometric information, avoiding feature smoothing or overfitting that may result from a single fusion operation, and enhancing the robustness of feature representation.

[0055] The CIFM module, through refined attention guidance and diverse fusion operators, successfully constructed a high-level bimodal feature representation with high discriminative power. This not only strengthens the consistency of cross-level semantics but also provides decisive semantic support for the final high-precision defect classification and localization.

[0056] Preferably, in S5, the CF module performs layer-by-layer fusion of multi-scale features in a top-down manner; in each fusion layer, the fusion features of the previous level are upsampled and then input into the channel attention mechanism and the foreground perception mechanism respectively to generate channel weights and foreground weights; the channel weights are multiplied with the multi-scale features of the current level, the foreground weights are multiplied with the multi-scale features of the current level, and the two multiplication results are added to the multi-scale features of the current level to obtain the enhanced features of the current level; wherein, the enhanced features of the lowest level are used as the final fusion features.

[0057] This setup achieves precise feature calibration through top-down semantic guidance and a dual-gating mechanism. By using a top-down fusion path, high-level features rich in semantic information are used as guiding signals. A channel attention mechanism is employed to filter out texture and geometric channels crucial for defect classification, while a foreground awareness mechanism explicitly distinguishes defect areas from complex backgrounds. This dual-gating strategy of "channel filtering + foreground focusing" effectively solves the problem of shallow features being easily interfered with by background noise in traditional FPN structures, ensuring that spatial details lost during downsampling can be accurately activated and corrected by high-confidence semantic information.

[0058] 2. Residual weighted fusion strategy ensures the integrity and robustness of multi-scale information. The residual fusion method, which involves multiplying the weights and then adding them to the original features, avoids the risk of feature suppression or gradient vanishing that may occur with traditional simple multiplication operations. This design strengthens the response of key defect features while preserving the complete distribution of the original multi-scale features. This allows the network to accurately capture small targets while maintaining the stability of the overall feature map when facing rail defects of different sizes and contrasts, significantly improving the model's multi-scale adaptability.

[0059] This CF module constructs an efficient and robust multi-scale feature pyramid through semantically driven dynamic weighting and residual connections, achieving seamless integration from high-level semantics to low-level details, and providing a feature foundation with both strong discriminative power and rich spatial information for the final high-precision detection head.

[0060] Preferably, in S6, the threshold used for thresholding is X, and pixels with a probability value greater than or equal to X in the defect prediction probability map are determined as defect areas; wherein, 0.4≤X≤0.6.

[0061] This setting, placing the threshold X within the range of 0.4 ≤ X ≤ 0.6, breaks the limitations of the traditional fixed threshold of 0.5, providing the model with a flexible decision boundary. In rail defect detection, facing the challenge of both weak cracks (low confidence) and complex background noise (high false alarm risk), this range allows for fine-tuning the strategy according to the actual application scenario: biasing towards 0.40 can reduce the false negative rate (improve the recall rate), ensuring that minor defects are not missed; biasing towards 0.6 can suppress false alarms (improve the precision rate), reducing false alarms caused by oil stains or reflections. This flexible setting effectively solves the contradiction between "better to miss a false positive than let one go" and "precise location" that a single threshold cannot satisfy.

[0062] Preferably, the surface defects of the rail include at least one of holes, scratches, and cracks.

[0063] This setup, by jointly modeling three vastly different morphological defects—holes (point-like / block-like), scratches (irregular surface-like), and cracks (thin, linear-like)—forces the network to simultaneously capture local texture abrupt changes, regional geometric anomalies, and long-range linear structures during feature extraction. This multi-objective setting avoids overfitting of the model to a single defect type, significantly improving the algorithm's generalization ability when facing complex and diverse damage on rail surfaces, ensuring comprehensive coverage of all potential safety hazards. Attached Figure Description

[0064] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0065] Figure 1 This is a flowchart of the method;

[0066] Figure 2 This is a schematic diagram of the detection network framework in Example 1;

[0067] Figure 3 This is a schematic diagram of the FDEM structure in Example 1;

[0068] Figure 4 This is a schematic diagram of the CIFM structure in Example 1;

[0069] Figure 5 This is a functional diagram of AU in Example 1;

[0070] Figure 6 This is a schematic diagram showing the results of validating the effectiveness of each model in Example 2. Detailed Implementation

[0071] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0073] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not indicate that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] Example 1

[0075] like Figure 1 As shown, this invention provides a rail defect detection method based on frequency information and a cross-modal fusion strategy, comprising the following steps:

[0076] S1. Preprocess the RGB-D rail image to generate a uniformly sized RGB image, a normalized depth image, and corresponding pixel-level binary mask labels.

[0077] In specific implementation, the preprocessing includes: uniformly adjusting the resolution of the RGB image and the depth image to 256×256 pixels, and normalizing the depth image. By uniformly adjusting the resolution of the RGB and depth images to 256×256 pixels and normalizing the depth image, this scheme effectively eliminates the heterogeneity of multimodal data in terms of spatial scale and numerical distribution. This preprocessing strategy not only ensures the consistency of the input dimension of the two-stream encoder, avoiding additional interpolation errors or alignment computation overhead caused by size mismatch, but also maps depth information to a stable numerical range, significantly accelerating the convergence process of the network gradient and improving training stability.

[0078] S2. Input the RGB image generated in S1 into the first encoder built on the SMT network, and input the depth image generated in S1 into the second encoder built on the MobileNetV2 network to extract RGB feature maps and depth feature maps of five levels respectively.

[0079] In specific implementation, the five extracted feature maps correspond to Stages 1 to 5, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original input image size, respectively. Both the SMT and MobileNetV2 networks are initialized using model parameters pre-trained on the ImageNet dataset. This initialization using SMT and MobileNetV2 models pre-trained on ImageNet, and the construction of a multi-scale feature pyramid from 1 / 4 to 1 / 64, effectively utilizes general texture and geometric prior knowledge learned from large-scale natural image data. This strategy significantly mitigates the risk of model overfitting due to the scarcity of industrial defect samples and accelerates network convergence. Simultaneously, the multi-level resolution feature representation ensures that the network can capture global semantic context through deep, low-resolution features while preserving the fine spatial structure of minute defects using shallow, high-resolution features, overcoming the limitation of single-scale features in balancing localization accuracy and classification accuracy. This design significantly improves the model's generalization performance and multi-scale defect adaptability under small sample conditions, providing a solid feature foundation for accurate detection in complex backgrounds.

[0080] S3, Frequency Domain Enhancement and Low-Level Fusion: For the RGB feature maps and depth feature maps of Stage 1 and Stage 2 in S2, perform the following sequentially:

[0081] 3a. Frequency Domain Dynamic Enhancement: Processing is performed through the frequency domain dynamic enhancement module FDEM. The FDEM module first generates a fused feature map based on the RGB feature map and the depth feature map, and performs discrete wavelet transform on the fused feature map to extract low-frequency and high-frequency components. Subsequently, using the low-frequency and high-frequency components, the original RGB feature map and the depth feature map are weighted and enhanced respectively through a gated dynamic fusion mechanism, and the enhanced RGB feature map and the enhanced depth feature map are output.

[0082] In practice, the FDEM module includes a fused feature map generation unit, a frequency domain decomposition unit, and a frequency domain fusion unit (FFU), used to dynamically enhance the RGB feature map and the depth feature map in the frequency domain. The process is as follows:

[0083] First, the upsampled RGB feature map and the upsampled depth feature map are concatenated by the fusion feature map generation unit, and a preliminary fusion feature map is obtained by 3×3 convolution.

[0084] Then, through the frequency domain decomposition unit, a discrete wavelet transform is performed on the preliminary fused feature map to extract its low-frequency and high-frequency components, and directional detail extraction is performed on the high-frequency components to obtain high-frequency information in the horizontal, vertical and diagonal directions, and then they are stitched together to form a multi-directional high-frequency feature map.

[0085] Finally, the low-frequency component and the multi-directional high-frequency feature map are dynamically fused with the original RGB feature map and depth feature map through the frequency domain fusion unit (FFU). The dynamic fusion includes: controlling the contribution ratio of low-frequency and high-frequency features using learnable weight parameters, and adding the weighted frequency domain features to the original feature map after spatial adjustment by 3×3 convolution to obtain the enhanced RGB feature map and depth feature map.

[0086] By decomposing features into low-frequency structures and multi-directional high-frequency components (horizontal, vertical, and diagonal) through discrete wavelet transform, this design overcomes the limitation of traditional spatial convolution in explicitly capturing textures in specific directions. For subtle defects with clear directions, such as cracks and scratches on rail surfaces, the extraction and utilization of multi-directional high-frequency information significantly amplifies their edge responses. Simultaneously, the low-frequency components maintain the stability of the background structure, effectively solving the problem of weak defect features being easily obscured under complex lighting conditions. Furthermore, learnable weight parameters are introduced to control the contribution ratio of frequency domain components to the original features, enabling the network to dynamically adjust the enhancement strategy based on the characteristics of the input content. Compared to feature addition or concatenation with fixed weights, this gating mechanism intelligently suppresses high-frequency noise interference such as metal reflections and oil stains, enhancing high-frequency details only when identified as defect areas. This improves the signal-to-noise ratio while avoiding artifacts caused by over-enhancement, achieving both accuracy and adaptability in feature enhancement. This module significantly improves the model's sensitivity to directional micro-defects and its noise robustness through reconstruction and dynamic modulation from a frequency domain perspective, providing high-purity feature inputs for subsequent cross-modal fusion.

[0087] 3b Low-level cross-modal fusion: The enhanced RGB feature map output from 3a and the deep feature map are input into the Interaction Unit (IIU) module for fusion. The IIU module first performs feature concatenation and enhancement on the two, then extracts spatial location-aware features based on the coordinate attention mechanism, and combines multi-scale spatial context information. Finally, it performs global weighted fusion through the spatial attention mechanism to output low-level fused features.

[0088] In specific implementation, the IIU module includes a stitching unit, a CBR structure, a coordinate attention mechanism, a depthwise separable convolution, multi-scale convolutional branches, and a spatial attention mechanism. It is used to interact and fuse the enhanced RGB feature map and the enhanced depth feature map output in step 3a. The process is as follows:

[0089] First, the enhanced RGB feature map and the enhanced depth feature map are concatenated to obtain the enhanced fused feature map;

[0090] Then, the concatenated enhanced fused feature map is subjected to feature enhancement processing through a CBR structure to improve the feature representation capability; the CBR structure includes convolutional layers, batch normalization, and ReLU activation function;

[0091] Next, spatial attention weights are generated through a coordinate attention mechanism and applied to the enhanced fusion feature map processed by the CBR structure to generate a spatially aware feature map.

[0092] Then, a 5×5 depthwise separable convolution is performed on the spatially perceived feature map to obtain a spatial relationship enhancement feature map;

[0093] Then, spatial correlation information at different scales is extracted by three convolutional branches of different scales, and the output of each branch is added element by element to the spatial relationship enhancement feature map to obtain a multi-scale fusion feature map.

[0094] Finally, a spatial attention mechanism is used to perform global spatial modeling on the multi-scale fused feature map, and low-level fused features are output.

[0095] By introducing a coordinate attention mechanism to embed positional information into channel attention, the network overcomes the limitation of traditional global pooling in losing spatial coordinates, enabling it to accurately pinpoint the specific location of rail defects. Combined with 5×5 depthwise separable convolutions, this effectively expands the receptive field to capture local spatial context while significantly reducing the number of parameters and computational complexity. This combination ensures that the model maintains real-time performance while possessing a strong ability to perceive and locate the spatial structure of minute defects. Furthermore, by setting three convolutional branches of different scales in parallel, the network can simultaneously capture multi-granular spatial correlation information, from fine textures to macroscopic morphology, solving the problem that single-scale convolutions cannot adapt to drastic changes in rail defect size. Subsequently, a spatial attention mechanism is used for global modeling to further suppress background noise and highlight key defect areas. This "multi-scale extraction + global optimization" strategy achieves deep fusion of RGB texture information and depth geometric information in the spatial dimension, significantly improving the robustness of defect detection in complex backgrounds. This module enhances the interaction efficiency of multimodal features through refined spatial attention guidance and efficient multi-scale feature aggregation. It also constructs a feature representation system that balances positioning accuracy, scale adaptability, and computational efficiency, providing high-quality low-level fusion features for subsequent high-level semantic reasoning.

[0096] S4. High-level feature cross-modal fusion: For the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2, the cross-interaction fusion module CIFM is used for fusion. The CIFM module first enhances the details of the bimodal features through the detail-aware unit DPU, and then extracts spatial location-aware features and channel context features based on spatial attention mechanism and coordinate attention mechanism respectively. The two are then integrated through multi-path interaction fusion strategy to output high-level bimodal features.

[0097] In practice, the CIFM module includes a detail-aware unit (DPU), a spatial attention mechanism (SA), a coordinate attention mechanism (CA), and a feature fusion unit. It is used to perform layer-by-layer high-level cross-modal fusion of RGB feature maps and depth feature maps from Stage 3 to Stage 5 in S2. The process is as follows:

[0098] First, the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2 are fused and enhanced with details by the detail-aware unit (DPU) to obtain the enhanced joint features of the corresponding levels.

[0099] Then, for each level, the enhanced joint features of that level are input into the spatial attention mechanism SA to generate spatial weights, which are then applied to the enhanced joint features of that level to obtain spatial enhanced features.

[0100] Next, for each level, the spatial augmentation features of that level are input into the coordinate attention mechanism (CA) to generate channel weights, which are then applied to the spatial augmentation features of that level to obtain channel augmentation features.

[0101] Then, for each level, the spatial enhancement features and channel enhancement features of that level are added and multiplied to obtain the first fusion feature and the second fusion feature, respectively.

[0102] Finally, the first and second fusion features of all levels are concatenated separately, and the channel dimensions are adjusted by 1×1 convolution to output the final high-level bimodal features.

[0103] This innovative approach employs a sequential processing strategy of first spatial attention (SA) and then coordinate attention (CA), forming a hierarchical filtering mechanism from "what is important" to "which features are important." SA first suppresses complex background noise and locks the spatial location of defects, while CA then uses coordinate information to enhance the response of specific channels to the texture and geometry of defects. This coarse-to-fine dual filtering effectively solves the problem of rich semantic information but blurred spatial details in high-level features, significantly improving the model's ability to discriminate small defects in strong interference backgrounds. Furthermore, by element-wise addition and multiplication of spatial enhancement features and channel enhancement features, a dual-path feature aggregation is constructed. The addition operation preserves the linear integrity of the original features, preventing the loss of key information; the multiplication operation introduces a strong nonlinear gating effect, forcing the network to focus on the high-confidence region where both respond together. This parallel fusion method of "preservation + enhancement" maximizes the complementary advantages of RGB texture information and depth geometric information, avoiding feature smoothing or overfitting that may be caused by a single fusion operation, and enhancing the robustness of feature representation. The CIFM module, through refined attention guidance and diverse fusion operators, successfully constructed a high-level bimodal feature representation with high discriminative power. This not only strengthens the consistency of cross-level semantics but also provides decisive semantic support for the final high-precision defect classification and localization.

[0104] S5. Cross-scale feature fusion: Construct a multi-scale feature representation composed of the low-level fusion features output by S3 and the high-level bimodal features output by S4. Perform top-down layer-by-layer fusion through the cross-scale fusion module CF. At each fusion layer, the CF module upsamples the fusion features of the previous level, generates multi-dimensional guiding weights based on the channel attention mechanism and the foreground perception mechanism, and uses the multi-dimensional guiding weights to perform weighted fusion of the multi-scale features of the current level to obtain enhanced features that fuse high-level semantics and low-level details. The enhanced features of the lowest level are used as the final fusion features.

[0105] In practice, the CF module fuses multi-scale features layer by layer in a top-down manner. At each fusion layer, the fusion features from the previous layer are upsampled and then input into the channel attention mechanism and the foreground perception mechanism to generate channel weights and foreground weights. The channel weights are multiplied by the multi-scale features of the current layer, and the foreground weights are multiplied by the multi-scale features of the current layer. The two multiplication results are then added to the multi-scale features of the current layer to obtain the enhanced features of the current layer. The enhanced features of the lowest layer are used as the final fusion features.

[0106] In this way, through a top-down fusion path, high-level features rich in semantic information are used as guiding signals. A channel attention mechanism is employed to filter out the texture and geometric channels crucial for defect classification, while a foreground awareness mechanism explicitly distinguishes defect regions from complex backgrounds. This dual-gating strategy of "channel filtering + foreground focusing" effectively solves the problem of shallow features being easily interfered with by background noise in traditional FPN structures, ensuring that spatial details lost during downsampling can be accurately activated and corrected by high-confidence semantic information. Furthermore, the residual fusion method of "multiplying weights and then adding them to the original features" avoids the risk of feature suppression or gradient vanishing that may result from traditional simple multiplication operations. This design strengthens the response to key defect features while preserving the complete distribution of the original multi-scale features. This allows the network to keenly capture small targets while maintaining the stability of the overall feature map when facing rail defects of different sizes and contrasts, significantly improving the model's multi-scale adaptability. The CF module constructs an efficient and robust multi-scale feature pyramid through semantically driven dynamic weighting and residual connections, achieving seamless connection from high-level semantics to low-level details, and providing a feature foundation with both strong discriminative power and rich spatial information for the final high-precision detection head.

[0107] S6. Defect Detection: The final fused features output from S5 are decoded to generate a defect prediction probability map. The probability map is then thresholded to output the location and range of defects on the rail surface.

[0108] The defects on the rail surface include at least one of holes, scratches, and cracks. Therefore, jointly modeling these three significantly different morphological defects—holes (point / block-like), scratches (irregular planar), and cracks (slender lines)—forces the network to simultaneously capture local texture abrupt changes, regional geometric anomalies, and long-range linear structures during the feature extraction stage. This multi-objective setting avoids overfitting the model to a single defect type, significantly improving the algorithm's generalization ability when facing complex and diverse damage to rail surfaces, and ensuring comprehensive coverage of various potential safety hazards.

[0109] In practical implementation, the thresholding process uses a threshold of X, and pixels with a probability value greater than or equal to X in the defect prediction probability map are identified as defect areas; where 0.4 ≤ X ≤ 0.6. Setting the threshold X in the range of 0.4 ≤ X ≤ 0.6 breaks the limitations of the traditional fixed threshold of 0.5, providing the model with a flexible decision boundary. In rail defect detection, facing the challenge of both weak cracks (low confidence) and complex background noise (high false alarm risk), this range allows for fine-tuning the strategy according to the actual application scenario: biasing towards 0.40 can reduce the false negative rate (improve the recall rate), ensuring that small defects are not missed; biasing towards 0.6 can suppress false alarms (improve the precision rate), reducing false alarms caused by oil stains or reflections. This flexible setting effectively solves the contradiction between "better to miss a false positive than let one go" and "precise location" that a single threshold cannot satisfy.

[0110] In specific implementation, steps S2 to S5 are executed by a detection network, which includes a dual-stream encoder, an FDEM module, an IIU module, a CIFM module, and a CF module, and is trained through end-to-end supervised learning. The training uses an RGB-D rail image dataset with pixel-level binary mask labels, and optimizes the network parameters using a joint loss function consisting of binary cross-entropy loss and cross-union ratio loss. During model training, data augmentation can be performed on the RGB-D images in the training set, including random horizontal flipping, vertical flipping, cropping, and brightness adjustment, to improve the model's generalization ability. Thus, by constructing an end-to-end trainable unified detection network, organically integrating the dual-stream encoder with multi-level fusion modules (FDEM, IIU, CIFM, CF), and optimizing using pixel-level label supervision and a joint loss function, collaborative learning of multimodal feature extraction, frequency domain enhancement, and cross-scale fusion is effectively achieved. Compared to traditional methods of staged training or manually designed fusion rules, this integrated architecture avoids information loss and error accumulation in intermediate stages, significantly improving the overall convergence stability and defect localization accuracy of the model.

[0111] To facilitate a better understanding of the detection network of S2-S5 by those skilled in the art, the following explanation is provided.

[0112] A schematic diagram of the detection network framework is shown below. Figure 2 As shown.

[0113] The acquired images are preprocessed, including adjusting the image resolution to a uniform 256×256 and using a data augmentation method of random flipping and random cropping. The annotation is performed by professional annotators who use annotation tools to delineate the defect areas at the pixel level and generate a real label image in the form of a binary mask. The defects include holes, scratches and cracks. The dataset used is a public dataset.

[0114] In S2, the first encoder built based on the SMT network and the second encoder built based on the MobileNetV2 network constitute a two-stream encoder. The two-stream encoder extracts RGB image features through a backbone network based on a transformer variant (SMT) and extracts depth image features through the MobileNetV2 network. Both are initialized with pre-trained parameters. The extracted multi-scale feature maps correspond to five levels, Stage 1 to Stage 5, with scales of 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original image, respectively. The feature maps are represented as follows: and ,in, For the RGB image Multi-scale feature maps of different layers For the depth image Multi-scale feature maps of different layers , , The first Width, height, and number of channels of the layer feature map.

[0115] In step 3a of S3, the structure of the FDEM module is as follows: Figure 3 As shown. The processing steps of the FDEM module include: adjusting by bilinear interpolation upsampling (up). and The dimensions and number of channels are then determined, and the resulting spliced ​​and fused feature maps are obtained. The calculation formula is as follows:

[0116] ;

[0117] Where up represents the upsampling operation. This indicates a splicing operation. It is a 3×3 convolution;

[0118] Extraction based on 1-level Discrete Wavelet Transform (DWT) low-frequency components and high-frequency component list Based on the direction detail extraction operation TR from The high-frequency information H in the horizontal direction, V in the vertical direction, and D in the diagonal direction are obtained by splitting the data and then concatenating them into a single high-frequency information. The operation is represented as follows:

[0119] ;

[0120] Based on the frequency domain fusion unit FFU respectively with and Gated dynamic fusion is performed to obtain the enhanced RGB feature map. and depth feature map The calculation formula is as follows:

[0121] ;

[0122] Among them, FFU stands for Frequency Domain Fusion Unit, which is used to adaptively fuse high and low frequency components in the frequency domain with the original modal features. These are learnable weight parameters for RGB image feature fusion. These are learnable weight parameters for deep image feature fusion. The original modal feature map is used when fusing RGB image features. for When fusing depth image features, for , For learnable parameters, It is the sigmoid activation function. This is a 3×3 convolution with a stride of 1 and the same padding method.

[0123] In step 3b, the processing procedure of the IIU module includes: splicing. and get After being processed by height and width average pooling, CBR structure convolutional layers, BN layers, and ReLU activation functions, it is generated through coordinate attention mechanism CA. and , and , Element-wise multiplication yields ⨂ Separable convolutions with a depth of 5×5 get By using three convolutional branches of different scales to... Spatial relationship extraction was performed using three convolutional methods: Branch1 with a 1×7 convolution, Branch2 with a 1×11 convolution, and Branch3 with a 1×21 convolution. These methods were used to extract spatial relationship information at different scales. Spatial relationships were extracted based on the convolutional branches and then compared with... Adding each element one by one yields ⨁ The spatial attention mechanism (SA) was used to obtain the result. The calculation formula is as follows:

[0124] .

[0125] In S4, the structure of CIFM is as follows: Figure 4 As shown. The CIFM module's processing procedure includes: and After element-wise addition (⊕), the result is processed by a cascaded 1×1 and 3×3 convolution (both with a stride of 1 and the same padding) and a DPU module. , Spatial weights are generated via SA, and... and Residual connection respectively ( Then, channel weights are generated via CA and multiplied element by element. ),get and , and The components are added and multiplied separately, then concatenated (Cat), and then convolved with a 1×1 convolution. (Step size 1, fill method same) Adjust the number of channels to obtain The calculation formula is as follows:

[0126] .

[0127] The DPU module captures high- and low-frequency information through multi-scale convolution and adds it to the original features to enhance details.

[0128] In S5, the processing steps of the CF module include: high-level features transposed convolution The scale and number of channels are adjusted by 1×1 convolution (all strides are 1, padding is the same). Channel weights are generated by CA, and foreground weights are generated by the AU module. The AU module is the original features minus average pooling features, which are then processed by the sigmoid activation function. Both are then combined with low-level features. Element-by-element multiplication ( ) after Residual connection obtained The calculation formula is as follows:

[0129] ;

[0130] The functions of the AU module are illustrated as follows: Figure 5 As shown.

[0131] Simultaneously, global information is extracted through max pooling, foreground-aware weights are generated and multiplied with the original features, then subjected to a 3×3 convolution. The enhanced features are obtained, and the calculation formula is as follows:

[0132] ;

[0133] in, is the sigmoid activation function, and Aug is the average pooling operation.

[0134] When building / training the detection network, Framework training was performed on a server equipped with an NVIDIA RTX 4060 GPU, with a batch size of 4, 200 training epochs, the Adam optimizer, and an initial learning rate of 5e−5. Generated by 3×3 convolution , These are the fused feature maps output from Stage 1 to Stage 5. for The corresponding saliency prediction map is trained under supervision using the total loss function, which is as follows:

[0135] ;

[0136] in, This is the ground truth map (a binary mask of labels defined at the pixel level). The binary cross-entropy loss function is... The intersection-union loss function is... The ablation experiments included comparative experiments on the removal of CIFM, FDEM and other modules, as well as ablation experiments on the sub-modules of FDEM and CIFM modules, to verify the effectiveness of each module.

[0137] The RGB-D rail image to be detected is sequentially processed through dual-modal feature extraction, frequency domain dynamic enhancement, high-low layer cross-modal fusion, detail perception and cross-scale fusion. The decoder outputs a defect prediction map in the form of a binary mask, and the specific location and range of the defects on the rail surface are determined by the prediction map.

[0138] Compared with existing technologies, this invention efficiently extracts dual-modal features by employing a dual-stream encoder composed of SMT and MobileNetV2. Combined with the collaborative design of FDEM, CIFM, IIU, DPU and CF modules, it optimizes the entire process of feature extraction and fusion. While retaining key defect information, it avoids redundant calculations and specifically addresses the contradiction between insufficient algorithm accuracy and large model size in existing technologies. It improves the accuracy of defect identification through detail perception and cross-scale fusion, and controls model complexity through modular and lightweight design. This achieves the effect of higher algorithm detection accuracy and lighter model size, making it more suitable for the actual application scenarios of rail inspection.

[0139] In addition, this invention uses the FDEM module to extract high and low frequency information in the image frequency domain using Haar wavelet transform, and combines the frequency domain features of RGB and depth images with a gated structure to enhance the difference features between the foreground and background from the frequency domain dimension. Furthermore, the foreground perception mechanism of the CF module suppresses interference from similar textures, specifically solving the problem of difficulty in distinguishing between rail surface defects and metallic luster due to similarity in color and texture in the existing technology. This allows the model to accurately capture defect edges and details, achieving a significant improvement in the distinction between the foreground and background, and even small defects can be accurately identified.

[0140] Furthermore, this invention employs a hierarchical cross-modal fusion strategy that integrates the IIU module and the CIFM module in parallel. The IIU module mines detailed information from low-level features, while the CIFM module captures semantic information from high-level features. By combining coordinate attention and spatial attention mechanisms, the complementary value of color information in RGB images and spatial distance information in depth images is fully exploited. This breaks down the fusion barriers between different modal features and effectively solves the problem of insufficient fusion caused by modal differences between RGB and depth images in existing technologies. It achieves the effect of deep complementarity of information from the two modalities and improves the reliability of defect detection.

[0141] This method can significantly improve the detection rate and anti-interference ability of small defects while ensuring efficient model deployment.

[0142] Example 2

[0143] To better illustrate the effectiveness of this method, the following comparative examples are provided.

[0144] Example 1

[0145] The rail defect detection system in Example 1 is built on the GFCNet network to address the difficulties in distinguishing between foreground and background and the insufficient fusion of RGB and depth images in rail defect detection. The system includes a data acquisition unit, a data processing unit, a feature extraction unit, a fusion module unit, a model training unit, and a defect detection unit. These units work together to complete the entire process from data input to defect identification. All calculation parameters are defined as follows:

[0146] Data acquisition unit

[0147] The data acquisition unit includes a first acquisition module and a second acquisition module, with parameter details as follows:

[0148] The first acquisition module uses an industrial camera (model: Baslerac A2500-14uc) to acquire RGB image data of the rails. The original resolution is uniformly 1920×1080, covering three types of defects on the rail surface: holes (600 images), scratches (550 images), and cracks (650 images), totaling 1800 RGB images, forming an RGB image dataset.

[0149] The second acquisition module uses a lidar (model: VelodyneVLP-16) to acquire depth image data at the corresponding location. The original resolution is the same as that of the RGB image (1920×1080), and the pixel value range is 0-10000 (unit: mm), forming a depth image dataset.

[0150] Dataset partitioning: The dataset was partitioned into a training set of 1260 images, a validation set of 360 images, and a test set of 180 images using a randomization algorithm in a 7:2:1 ratio.

[0151] Data augmentation parameters: The training set uses random horizontal flipping (probability 0.5), random vertical flipping (probability 0.5), random cropping (cropped size 224×224, then resized to 256×256), and brightness adjustment (brightness variation range ±15%). After augmentation, the training set size reaches 2520 images.

[0152] Data processing unit

[0153] The data processing unit includes a preprocessing module and a labeling module, with parameter details as follows:

[0154] Preprocessing module: The resolution of both the RGB and depth images is uniformly adjusted to 256×256; the depth image undergoes min-max normalization (pixel values ​​are mapped to the [0,1] interval), using the following formula: ,in For the minimum depth, This represents the maximum depth.

[0155] Labeling module: Professional labelers use the labelimg tool to define the defect area at the pixel level and generate a ground truth map in the form of a binary mask. The pixel value of the defect area is 1 and the background area is 0. The labeling accuracy is ≥99%.

[0156] Feature extraction unit

[0157] The feature extraction unit is a two-stream encoder structure, and the parameter details are as follows:

[0158] RGB feature extraction module: Employs a backbone network based on a transformer variant (SMT), with pre-trained parameters derived from the ImageNet dataset, extracting feature maps from five levels, from Stage 1 to Stage 5. ,in:

[0159] Stage 1 (Scale 1 / 4): =64、 =64、 =64;

[0160] Stage 2 (1 / 8 scale): =32、 =32、 =128;

[0161] Stage 3 (1 / 16 scale): =16、 =16、 =256;

[0162] Stage 4 (1 / 32 scale): =8、 =8、 =512;

[0163] Stage 5 (Scale 1 / 64): =4、 =4、 =1024;

[0164] Deep feature extraction module: Employs the MobileNetV2 network, with pre-trained parameters derived from the ImageNet dataset, and simultaneously extracts deep feature maps from five layers. The number of channels and the size are completely consistent with the RGB feature map, ensuring scale and dimension matching.

[0165] Fusion module unit

[0166] The fusion module unit includes the FDEM module, IIU module, CIFM module, and CF module. Parameter details are as follows:

[0167] FDEM module (for low-level features from Stage 1 to Stage 2):

[0168] Upsampling parameters: Bilinear interpolation upsampling, which will... and The dimensions are adjusted to the original scale (Stage 1: 64×64, Stage 2: 32×32), while the number of channels remains unchanged (Stage 1: 64, Stage 2: 128).

[0169] Concatenation and Convolution: The number of channels after concatenation is ×2 (Stage 1: 128, Stage 2: 256), 3×3 convolution (stride 1, padding same, number of kernels the same as the number of channels after concatenation, bias term True), to obtain the fused feature map. The calculation formula is: ;

[0170] Discrete Wavelet Transform (DWT): 1-level Haar wavelet transform, decomposing low-frequency components. Channel number and Consistent (Stage 1: 128, Stage 2: 256), High-Frequency Component List It includes three directions: horizontal (H), vertical (V), and diagonal (D). The number of channels in each direction is... Consistent, after splicing The number of channels is ×2×3 (Stage1: 384, Stage2: 768);

[0171] Frequency Fusion Unit (FFU): Learnable Parameters =0.4 (RGB feature fusion) =0.6 (deep feature fusion), COV is a 3×3 convolution (stride 1, padding method same, number of convolution kernels same as...). (Consistent, bias term is True), the sigmoid activation function has no additional parameters, and the calculation formula is:

[0172] ;

[0173] IIU module (for enhanced features from Stage 1 to Stage 2):

[0174] splicing and CBR structure: and The number of channels after splicing is ×2 (Stage 1: 128, Stage 2: 256), CBR structure with 1×1 convolution (stride 1, padding same, number of kernels 128 / 256, bias True), BN layer (momentum 0.9, epsilon=1e-5), ReLU activation;

[0175] Coordinate Attention (CA): Average pooling in the height / width directions (pooling kernel sizes are respectively) ×1、1× Fully connected layer 1 (number of neurons) ×2 / / 16, Stage 1: 8, Stage 2: 16), ReLU activation, fully connected layer 2 (number of neurons) ×2, Stage 1: 128, Stage 2: 256), generated and (The number of channels is consistent with the characteristics after splicing);

[0176] Depthwise Separable Convolution (DWConv): 5×5 depthwise separable convolution, depthwise convolution (stride 1, same padding, number of groups = number of input channels), pointwise convolution (stride 1, same padding, number of kernels = number of input channels), output... ;

[0177] Multi-scale convolution branches: Branch1 (1×7 convolution, stride 1, padding 3), Branch2 (1×11 convolution, stride 1, padding 5), Branch3 (1×21 convolution, stride 1, padding 10), all are depthwise separable convolutions, with the number of output channels equal to... Consistent;

[0178] Spatial Attention (SA): 3×3 convolution (stride 1, padding same, kernel size 1), sigmoid activation, generating attention weight mask (size... × ),and Element-wise multiplication yields ;

[0179] CIFM module (for Stage 3 to Stage 5 high-level features):

[0180] Feature summation and convolution concatenation: and After element-wise addition, a 1×1 convolution (stride 1, padding same, number of kernels) is performed. Stage 3: 256, Stage 4: 512, Stage 5: 1024), 3×3 convolution (stride 1, padding same, number of kernels) Number of output channels ;

[0181] DPU module: Multi-scale convolution (1×3, 3×1, 3×3, stride 1, padding same, number of kernels) Add the original features element by element to the output. ;

[0182] Spatial Attention (SA): 3×3 convolution (stride 1, padding same, number of kernels 1), sigmoid activation, generating spatial weights; Coordinate Attention (CA) parameters are the same as the IIU module, generating channel weights;

[0183] Final Fusion: and Add, multiply, and then concatenate (number of channels 2 × ), 1×1 convolution (stride 1, padding same, number of kernels) Output the final high-level fusion features ;

[0184] CF module (decoder):

[0185] Transposed convolution and 1×1 convolution: Transposed convolution (4×4 kernel, stride 2, padding 1, output size 2×1) ), 1×1 convolution (stride 1, padding same, number of kernels Ci), adjust high-level features Down to the lowest feature scale;

[0186] Coordinate Attention (CA) and AU Module: CA generates channel weights (number of channels) In the AU module, average pooling (global average pooling), the original feature is subtracted from the average pooling feature, and sigmoid activation is used to generate foreground weights (number of channels). );

[0187] Enhanced feature calculation: 3×3 convolution (stride 1, padding same, number of kernels) ), sigmoid activation, foreground-aware weights are multiplied element-wise with the original features to output enhanced features. ;

[0188] Model training unit

[0189] The training unit is built on the PyTorch framework, and the hardware environment is an NVIDIA RTX 4060 GPU (8GB VRAM). Parameter details are as follows:

[0190] Training parameters: batch size = 4, training epochs = 200, initial learning rate = 5e-5, optimizer = Adam (betas = (0.9, 0.999), epsilon = 1e-8), weight decay coefficient = 1e-4;

[0191] Learning rate scheduling: Cosine annealing scheduling is used (maximum number of iterations in cosine annealing scheduling). =200, minimum learning rate for cosine annealing scheduling =1e-6), no warm-up rounds;

[0192] Loss function: Weighted sum of Binary Cross-Entropy (BCE) loss function (weight=1.0) and Cross-Intersection over Union (IOU) loss function (weight=1.0) to obtain the total loss. ,in The saliency prediction map is generated by convolving the feature maps of Stage 1 to Stage 4 with 3×3 (stride 1, padding same, number of kernels 1);

[0193] Early stopping strategy: Stop training when the validation set loss does not decrease for 15 consecutive epochs (decline threshold <1e-4), save the optimal model weights, and save the model in . / best_model.pth;

[0194] Defect detection unit

[0195] The testing process parameters are as follows:

[0196] Input preprocessing: The resolution of the RGB-D image to be detected is adjusted to 256×256 according to the training set preprocessing standard, and the depth image is normalized;

[0197] Post-processing: The decoder outputs a binary mask (threshold = 0.5, pixel values ​​≥ 0.5 are judged as defects), non-maximum suppression (IOU threshold = 0.5, confidence threshold = 0.5), and filters redundant defect areas;

[0198] Output results: Defect category (hole, scratch, crack), bounding box coordinates (calculated based on the bounding rectangle of the binary mask), confidence score (pixel percentage of defect area × model output probability).

[0199] Example 2

[0200] Example 2 has the same core structure as Example 1, with only some module parameters adjusted. The specific differences are as follows:

[0201] The multi-scale convolutional branches of the IIU module have been adjusted to 1×5 (Branch1), 1×9 (Branch2), and 1×17 (Branch3) to adapt to the extraction of defect features at a smaller scale.

[0202] The initial learning rate was adjusted to 4e-5, the weight decay coefficient was adjusted to 5e-5, and the optimizer was AdamW (betas=(0.9,0.999), epsilon=1e-8).

[0203] In the FDEM module, the number of decomposition layers of the Haar wavelet transform is adjusted to 2 layers to further refine the frequency domain feature extraction. After decomposition, the number of channels in the low-frequency components remains unchanged, while the number of channels in each direction of the high-frequency components is split into 1 / 2 of the original number of channels. After splicing, the total number of channels is consistent with that in Example 1.

[0204] The data augmentation methods include new brightness adjustment (brightness variation range ±15%) and Gaussian blur (kernel size 3×3, standard deviation 0.5) to improve the model's generalization ability. The remaining module composition, processing steps, and training / detection process are completely consistent with Example 1.

[0205] • Comparative

[0206] Patent application number: 2025110179073;

[0207] Patent title: Training method for rail defect detection model and detection method for rail defects;

[0208] System structure and processing flow:

[0209] This comparative example is strictly based on the technical solution of the aforementioned patent. Its core is an improved YOLOv8 model. It does not employ RGB-D dual-modal data, frequency domain enhancement, or hierarchical cross-modal fusion strategies, as detailed below:

[0210] Data acquisition: Only RGB single-modal images were used. The dataset was the same as in Example 1 (1800 images, 600 images of holes, 550 images of scars, and 650 images of cracks). The dataset was divided into a training set of 1260 images, a validation set of 360 images, and a test set of 180 images in a 7:2:1 ratio. No depth image acquisition and processing were performed.

[0211] Data preprocessing: The image resolution was adjusted to 256×256, and data augmentation was performed using random flipping (probability 0.5) and random cropping (224×224 resized to 256×256), without brightness adjustment or Gaussian blur; the annotation method was the same as in Example 1, generating binary mask labels;

[0212] Feature extraction: An improved YOLOv8 backbone network is used. The feature extraction layer includes convolutional layers, cross-stage fusion layers (C2f layers), and channel enhancement layers (SaE modules). SaE modules are embedded at the end of each BottleNeck of the C2f modules to extract multi-scale feature maps (Stage 1 to Stage 5, scales 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64) of the single-modality RGB image. There is no dual-stream encoder structure.

[0213] Feature fusion: Through the CCFF cross-scale fusion mechanism, 3×3 and 1×1 scale information is extracted in parallel based on reparameterized convolution (RepConv). The number of channels is adjusted by two 1×1 convolutions (input channel number / / 2). After feature fusion by three RepBlocks, the outputs of the two paths are added element-wise to achieve fusion. There are no FDEM, IIU, or CIFM modules, and frequency domain feature extraction and gated fusion are not involved.

[0214] Model training: The training hardware is the same as in Example 1 (NVIDIA RTX 4060 GPU). Training parameters: batch size = 4, training epochs = 200, initial learning rate = 5e-5, optimizer = Adam, weight decay coefficient = 1e-4; the loss function adopts the WIoUv3 loss function, and the non-monotonic focusing coefficient is calculated by the monotonic focusing coefficient. The target loss is obtained by combining the intersection-over-union loss and the center distance loss. There is no weighted combination of BCE and IOU loss functions.

[0215] Defect detection: Input a single-modal RGB image, and after feature extraction, cross-scale fusion, prediction box generation and non-maximum suppression (IOU threshold=0.5), output the defect detection result. No depth information is used to assist in the judgment. The model deployment can be converted to ONNX format and the inference speed can be optimized by the TensorRT engine.

[0216] Summary of Effects

[0217] Comparison metrics and testing environment

[0218] The testing environment was the same as the training environment (NVIDIA RTX 4060 GPU, 8GB VRAM), using the test set of Example 1 (180 images, including three types of defects). The comparison metrics included mean accuracy (mAP@0.5), number of model parameters (Params), inference speed (FPS), foreground / background discrimination accuracy (measured by the mean IOU between the defect region and the background region), and individual detection accuracy of the three types of defects (AP@0.5).

[0219] The comparison results are shown in Table 1 below:

[0220] Table 1

[0221]

[0222] Results Analysis

[0223] Detection accuracy: The mAP@0.5 of Examples 1 and 2 is 8.8% and 9.6% higher than that of the comparative example, respectively, and the individual detection accuracy of the three types of defects is 8%-12% higher. The core reason is that Examples 1 and 2 use RGB-D dual-modal data, extract frequency domain features through the 1st / 2nd layer Haar wavelet transform of the FDEM module, and enhance the frequency domain difference between the foreground and background by combining a gated fusion strategy. At the same time, the hierarchical cross-modal fusion of the IIU and CIFM modules fully explores the complementary value of color information (RGB) and spatial distance information (depth), solving the problem of insufficient feature expression of single-modal data. In contrast, the comparative example only relies on RGB single-modal data and is only optimized by the C2f-SaE module and CCFF cross-scale fusion. It lacks frequency domain enhancement and cross-modal fusion, making it difficult to distinguish the similar features of rail metallic luster and defect texture.

[0224] Lightweight Model: The number of parameters in Examples 1 and 2 is reduced by 33.3% and 31.0% respectively compared to the comparison, while the inference speed is improved by 28.2% and 20.3% respectively. Thanks to the lightweight design of the FDEM and IIU modules (deepestiable separable convolution and efficient feature selection through attention mechanisms), and the hierarchical fusion strategy to reduce the propagation of invalid features, redundant computations of heavily parameterized convolutions and C2f-SaE modules in the comparison are avoided, thus reducing model complexity while maintaining accuracy.

[0225] Foreground / background differentiation capability: The differentiation accuracy of Examples 1 and 2 is more than 13% higher than that of the comparison. The key is that the FDEM module amplifies the difference between defects and background in the frequency domain dimension through frequency domain feature extraction and gated dynamic fusion. In contrast, the comparison only relies on channel attention and cross-scale fusion, which cannot effectively solve the differentiation problem caused by similar color and texture. It is especially outstanding in the detection of subtle defects such as cracks.

[0226] Further explanation is needed regarding the ablation experiments and the proof of their necessity:

[0227] Impact of different modules: To verify the importance of each module, we conducted ablation experiments on them. As shown in Table 2, we designed six experiments: (a) BaseLine: using only feature addition and convolution operations, (b) w / o CIFM: replacing with element-wise summation operations, (c) w / o FDEM: replacing with convolution and element-wise summation operations, (d) w / o DPU, (e) w / o CF: replacing with convolution and element-wise summation operations, and (f) Ours.

[0228] Table 2 - Effectiveness Analysis of Different Modules

[0229]

[0230] Qualitative comparison of ablation experiments using different modules is shown in the figure below. Figure 6 As shown, compared with the baseline model, adding FDEM and CF modules improved the model's ability to distinguish between foreground and background, and adding CIFM and DPU modules improved the model's detection accuracy, proving its effectiveness.

[0231] Table 3 provides a unified definition for some of the symbols appearing in the instruction manual:

[0232] Table 3

[0233]

[0234] In summary, the rail defect detection method based on frequency information and cross-modal fusion strategy of the present invention significantly outperforms existing technologies in terms of detection accuracy, model lightweighting, foreground-background differentiation ability, and inference speed through innovative design of frequency domain dynamic enhancement of FDEM module, hierarchical cross-modal fusion of IIU and CIFM, and detail perception and cross-scale fusion of DPU and CF. It is more suitable for the actual industrial needs of rail defect detection, and is especially suitable for the identification of subtle defects in complex scenarios.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A rail defect detection method based on frequency information and a cross-modal fusion strategy, characterized in that, Includes the following steps: S1. Preprocess the RGB-D rail image to generate a uniformly sized RGB image, a normalized depth image, and corresponding pixel-level binary mask labels. S2. Input the RGB image generated in S1 into the first encoder built on the SMT network, and input the depth image generated in S1 into the second encoder built on the MobileNetV2 network to extract RGB feature maps and depth feature maps at five levels respectively. S3, Frequency Domain Enhancement and Low-Level Fusion: For the RGB feature maps and depth feature maps of Stage 1 and Stage 2 in S2, perform the following sequentially: 3a Frequency Domain Dynamic Enhancement: Processed by the frequency domain dynamic enhancement module FDEM. The FDEM module first generates a fused feature map based on the RGB feature map and the depth feature map, and then performs discrete wavelet transform on the fused feature map to extract low-frequency and high-frequency components. Subsequently, using the low-frequency and high-frequency components, the original RGB feature map and depth feature map are weighted and enhanced respectively through a gated dynamic fusion mechanism, and the enhanced RGB feature map and enhanced depth feature map are output. 3b Low-level cross-modal fusion: The enhanced RGB feature map output from 3a is fused with the depth feature map input to the Interaction Unit (IIU) module; The IIU module first concatenates and enhances the features of the two, then extracts spatial location-aware features based on the coordinate attention mechanism, and combines multi-scale spatial context information. Finally, it performs global weighted fusion through the spatial attention mechanism to output low-level fused features. S4. High-level feature cross-modal fusion: For the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2, the cross-interaction fusion module CIFM is used for fusion. The CIFM module first enhances the details of the bimodal features through the detail-aware unit DPU, and then extracts spatial position-aware features and channel context features based on spatial attention mechanism and coordinate attention mechanism respectively. The two are then integrated collaboratively through multi-path interaction fusion strategy to output high-level bimodal features. S5. Cross-scale feature fusion: Construct a multi-scale feature representation composed of the low-level fusion features output by S3 and the high-level bimodal features output by S4. Perform top-down layer-by-layer fusion through the cross-scale fusion module CF. At each fusion layer, the CF module upsamples the fusion features of the previous level, generates multi-dimensional guiding weights based on the channel attention mechanism and the foreground perception mechanism, and uses the multi-dimensional guiding weights to perform weighted fusion of the multi-scale features of the current level to obtain enhanced features that fuse high-level semantics and low-level details. The lowest-level enhanced features are used as the final fused features; S6. Defect Detection: The final fused features output from S5 are decoded to generate a defect prediction probability map. The probability map is then thresholded to output the location and range of defects on the rail surface.

2. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in any one of claims 1, characterized in that, Steps S2 to S5 are performed by a detection network, which includes a dual-stream encoder, an FDEM module, an IIU module, a CIFM module, and a CF module, and is trained through end-to-end supervised training. The training uses an RGB-D rail image dataset with pixel-level binary mask labels, and optimizes the network parameters using a joint loss function consisting of binary cross-entropy loss and cross-union ratio loss.

3. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In S1, the preprocessing includes: uniformly adjusting the resolution of the RGB image and the depth image to 256×256 pixels, and normalizing the depth image.

4. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In S2, the five extracted layer feature maps correspond to Stage 1 to Stage 5, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original input image size, respectively; both the SMT network and the MobileNetV2 network are initialized using model parameters pre-trained on the ImageNet dataset.

5. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In step 3a of S3, the FDEM module includes a fused feature map generation unit, a frequency domain decomposition unit, and a frequency domain fusion unit (FFU), which is used to dynamically enhance the RGB feature map and the depth feature map in the frequency domain. The process is as follows: First, the upsampled RGB feature map and the upsampled depth feature map are concatenated by the fusion feature map generation unit, and a preliminary fusion feature map is obtained by 3×3 convolution. Then, through the frequency domain decomposition unit, a discrete wavelet transform is performed on the preliminary fused feature map to extract its low-frequency and high-frequency components, and directional detail extraction is performed on the high-frequency components to obtain high-frequency information in the horizontal, vertical and diagonal directions, and then they are stitched together to form a multi-directional high-frequency feature map. Finally, the low-frequency component and the multi-directional high-frequency feature map are dynamically fused with the original RGB feature map and depth feature map through the frequency domain fusion unit (FFU). The dynamic fusion includes: controlling the contribution ratio of low-frequency and high-frequency features using learnable weight parameters, and adding the weighted frequency domain features to the original feature map after spatial adjustment by 3×3 convolution to obtain the enhanced RGB feature map and depth feature map.

6. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In step 3b of S3, the IIU module includes a concatenation unit, a CBR structure, a coordinate attention mechanism, a depthwise separable convolution, a multi-scale convolution branch, and a spatial attention mechanism. It is used to interact with and fuse the enhanced RGB feature map and the enhanced depth feature map output from step 3a. The process is as follows: First, the enhanced RGB feature map and the enhanced depth feature map are concatenated to obtain the enhanced fused feature map; Then, the concatenated enhanced fused feature map is subjected to feature enhancement processing through a CBR structure to improve the feature representation capability; the CBR structure includes convolutional layers, batch normalization, and ReLU activation function; Next, spatial attention weights are generated through a coordinate attention mechanism and applied to the enhanced fusion feature map processed by the CBR structure to generate a spatially aware feature map. Then, a 5×5 depthwise separable convolution is performed on the spatially perceived feature map to obtain a spatial relationship enhancement feature map; Then, spatial association information at different scales is extracted by three convolutional branches of different scales, and the output of each branch is added element by element to the spatial relationship enhancement feature map to obtain a multi-scale fusion feature map; Finally, a spatial attention mechanism is used to perform global spatial modeling on the multi-scale fused feature map, and low-level fused features are output.

7. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In S4, the CIFM module includes a detail-aware unit (DPU), a spatial attention mechanism (SA), a coordinate attention mechanism (CA), and a feature fusion unit. It is used to perform layer-by-layer high-level cross-modal fusion of RGB feature maps and depth feature maps from Stage 3 to Stage 5 in S2. The process is as follows: First, the RGB feature maps and depth feature maps of Stage 3, Stage 4 and Stage 5 in S2 are fused and enhanced with details by the detail-aware unit (DPU) to obtain the enhanced joint features of the corresponding levels. Then, for each level, the enhanced joint features of that level are input into the spatial attention mechanism SA to generate spatial weights, which are then applied to the enhanced joint features of that level to obtain spatial enhanced features. Next, for each level, the spatial augmentation features of that level are input into the coordinate attention mechanism (CA) to generate channel weights, which are then applied to the spatial augmentation features of that level to obtain channel augmentation features. Then, for each level, the spatial enhancement features and channel enhancement features of that level are added and multiplied to obtain the first fusion feature and the second fusion feature, respectively. Finally, the first and second fusion features of all levels are concatenated separately, and the channel dimensions are adjusted by 1×1 convolution to output the final high-level bimodal features.

8. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In S5, the CF module fuses multi-scale features layer by layer in a top-down manner; In each fusion layer, the fusion features from the previous layer are upsampled and then input into the channel attention mechanism and the foreground perception mechanism to generate channel weights and foreground weights. The channel weights are multiplied by the multi-scale features of the current layer, and the foreground weights are multiplied by the multi-scale features of the current layer. The two multiplication results are then added to the multi-scale features of the current layer to obtain the enhanced features of the current layer. The enhanced features of the lowest layer are used as the final fusion features.

9. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, In S6, the thresholding process uses a threshold of X, and pixels with a probability value greater than or equal to X in the defect prediction probability map are identified as defect areas; where 0.4≤X≤0.

6.

10. The rail defect detection method based on frequency information and cross-modal fusion strategy as described in claim 1, characterized in that, The surface defects of the rail include at least one of holes, scratches, and cracks.

Images