RGB-D steel rail defect segmentation method and system based on prior guidance and channel fusion

Abstract

The invention provides an RGB-D steel rail defect segmentation method and system based on prior guidance and channel fusion. The method comprises the steps that an RGB image and a depth image of the surface of a steel rail are acquired; extracting multi-scale heterogeneous global features of the RGB image and the depth image, and dividing the multi-scale heterogeneous global features into low-level features and high-level features; performing prior-guided space fusion on the low-layer features to obtain low-layer fusion features; performing channel adaptive fusion on the high-level features to obtain high-level fusion features; and decoding the low-layer fusion features and the high-layer fusion features to obtain a defect segmentation map of the surface of the steel rail. According to the method, the technical means of combining a priori space attention mechanism and channel adaptive fusion is adopted, and the technical effect of high-precision positioning and segmentation of tiny defects can be achieved.

Description

Technical Field

[0001] The present application relates to the field of industrial detection technology, and in particular to an RGB-D rail defect segmentation method and system based on prior guidance and channel fusion. Background Art

[0002] Railway surface defect detection is a key technology in the field of railway operation and maintenance. Its core challenges stem from the special physical properties of the railway surface and the complexity of the detection scene. The railway surface presents a periodically repeated regular texture. This uniform background feature forms low-contrast interference with the tiny defect area, making it difficult for traditional background modeling methods to effectively distinguish defects from the background. For example, general methods such as Gaussian mixture models often produce significant false detections in railway scenes because they cannot adapt to the periodicity of textures, and their ability to suppress uniform texture interference is seriously insufficient. In addition, the railway surface is exposed to complex environments for a long time. Factors such as lighting changes, oil adhesion and surface oxidation will further aggravate the ambiguity of defect boundaries, posing severe challenges to traditional edge detection algorithms. Since the geometric contours of linear defects (such as cracks) usually present discontinuous and weak edge characteristics, their gradient response intensity is often at the same order of magnitude as the background noise, resulting in severe fragmentation of edge extraction results, which makes it difficult to meet the requirements of industrial inspection for defect contour integrity.

[0003] Existing detection technologies have some technical bottlenecks: First, multimodal data is not fully utilized. Most methods rely only on a single RGB modality or adopt simple feature splicing strategies, failing to fully tap the complementarity between RGB texture features and deep geometric features. Traditional feature splicing methods often ignore the semantic differences between modalities, which often leads to semantic misalignment of high-level features and introduces additional fusion noise. Second, there is a lack of high-level semantic fusion mechanisms. High-order features have poor cross-modal fusion effects due to channel redundancy and semantic differences. Traditional fusion methods will significantly reduce the signal-to-noise ratio of high-level features, which directly affects the accuracy of defect location. These limitations make it difficult for existing technologies to meet the comprehensive requirements of industrial inspection for accuracy, speed, and robustness. Summary of the Invention

[0004] In response to one of the defects in the prior art, the purpose of this application is to provide an RGB-D rail defect segmentation method and system based on prior guidance and channel fusion.

[0005] In a first aspect of the present application, a method for RGB-D rail defect segmentation based on prior guidance and channel fusion is provided, comprising:

[0006] Obtain RGB images and depth images of the rail surface;

[0007] Extracting multi-scale heterogeneous global features of the RGB image and the depth image, and dividing them into low-level features and high-level features;

[0008] Performing a priori-guided spatial fusion on the low-level features to obtain low-level fused features;

[0009] Performing channel adaptive fusion on the high-level features to obtain high-level fused features;

[0010] A decoding operation is performed on the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface.

[0011] Optionally, the extracting multi-scale heterogeneous global features of the RGB image and the depth image, dividing them into low-level features and high-level features, includes:

[0012] A dual-stream encoder is used to extract RGB features and depth features respectively;

[0013] The RGB features are fed into the RGB branch, and multiple scale RGB feature maps are output through a multi-layer feature extraction network constructed by RMT;

[0014] The depth features are fed into the depth branch, and multiple scale depth feature maps are output through a multi-layer feature extraction network constructed by MobileNetV2;

[0015] A convolutional layer is used to unify the feature dimensions of the multiple scale RGB feature maps and the multiple scale depth feature maps, and group them into low-level features and high-level features according to their scale sizes.

[0016] Optionally, performing a priori-guided spatial fusion on the low-level features to obtain low-level fused features includes:

[0017] Background suppression is performed on the RGB features in the low-level features, and multi-scale features are extracted using group convolution with different dilation rates. The multi-scale features are activated by instance normalization and Gaussian error linear unit respectively, and the activated multi-scale features are channel-joined. The first spatial attention mask weight is generated through a convolution layer and Sigmoid activation;

[0018] For the RGB features in the low-level features, the local variance is calculated using a fixed-size sliding window, and the second spatial attention mask weight is generated through variance normalization and Sigmoid activation;

[0019] Calculating the gradient magnitudes of the RGB features and the depth features using horizontal and vertical separable gradients, and generating the third spatial attention mask weight and the fourth spatial attention mask weight respectively through strength normalization, convolution, and Sigmoid activation;

[0020] The RGB features in the low-level features are element-wise multiplied with the first, second, and third spatial attention mask weights respectively, and the depth features are element-wise multiplied with the fourth spatial attention mask weights, and then residuals are added with the RGB features and the depth features to obtain the enhanced features after fusion at this level, i.e., the low-level fusion features.

[0021] Optionally, performing channel adaptive fusion on the high-level features to obtain high-level fused features includes:

[0022] Grouping the RGB features and the depth features of the high-level features into local refinement subgroups and global semantic subgroups respectively;

[0023] Applying depthwise separable convolution to the local refined subgroups of RGB features and depth features to enhance details;

[0024] For the global semantic subgroup and the local refined subgroup after detail enhancement, cross-channel splicing of two sets of cross-modal features is performed to obtain two sets of fused convolutions;

[0025] Shuffling the channels of the two groups of fused convolutions and performing point-by-point convolution to obtain a first group of enhanced features and a second group of enhanced features;

[0026] Perform global average pooling and two fully connected layers and Softamax on the first set of enhanced features and the second set of enhanced features to obtain channel attention weights;

[0027] The channel attention weights are respectively multiplied by the first set of enhanced features and the second set of enhanced features at the channel level, and the enhanced features are element-wise added to finally obtain the high-level fused features, i.e., the high-level fused features.

[0028] Optionally, the decoding operation on the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface includes:

[0029] Arrange the low-level and high-level fusion features in descending order of scale;

[0030] The highest-level fusion features are refined through a re-parameterized depth-wise separable convolution module;

[0031] The refined features are upsampled and concatenated with the previous level features to obtain new level features;

[0032] Repeat the above process twice to obtain a feature map of the same size as the first-level feature;

[0033] The feature map obtained above is interpolated and upsampled to obtain the final surface defect segmentation map.

[0034] Optionally, during the training phase, each of the re-parameterized depth-wise separable convolutions includes three summed parts, namely: a 3×3 depth-wise separable convolution, a 1×1 depth-wise separable convolution, and the original feature residual;

[0035] During the inference phase: each of the re-parameterized depth-wise separable convolutions is a 3×3 depth-wise separable convolution.

[0036] Optionally, the training process of the rail defect segmentation model corresponding to the segmentation method includes:

[0037] Determine a training sample set, including samples of rail surface defect RGB images and depth images and their corresponding segmentation ground truth maps;

[0038] Inputting the samples into the initial segmentation model respectively to obtain the segmentation results output by the initial segmentation model;

[0039] Calculating a binary cross entropy loss and an intersection-over-union loss based on the segmentation result and the segmentation truth map, and calculating a total loss based on the binary cross entropy loss and the intersection-over-union loss;

[0040] Based on the total loss, the alternating direction multiplier method is used to iteratively update the structural parameters of the initial segmentation model to obtain the rail defect segmentation model.

[0041] The second aspect of the present application provides an RGB-D rail defect segmentation system based on prior guidance and channel fusion, comprising:

[0042] Image acquisition module, which acquires RGB images and depth images of the rail surface;

[0043] A global heterogeneous extraction module extracts multi-scale heterogeneous global features of the RGB image and the depth image and divides them into low-level features and high-level features;

[0044] A priori guided module performs a priori guided spatial fusion on the low-level features to obtain low-level fusion features;

[0045] A channel adaptive module performs channel adaptive fusion on the high-level features to obtain high-level fusion features;

[0046] The defect segmentation module decodes the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface.

[0047] According to a third aspect of the present application, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the method described, or to run the system described, when executing the program.

[0048] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the method described, or run the system described.

[0049] The RGB-D rail defect segmentation method based on prior guidance and channel fusion provided in this application adopts a technical means combining the prior spatial attention mechanism with channel adaptive fusion, which brings the technical effect of achieving high-precision positioning and segmentation of tiny defects.

[0050] Other technical effects brought about by the additional features will be further explained in the corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0052] Figure 1 3 is a flowchart of an RGB-D rail defect segmentation method based on prior guidance and channel fusion according to an exemplary embodiment;

[0053] Figure 2 is a flowchart of heterogeneous feature extraction according to an exemplary embodiment;

[0054] Figure 3 1 is a schematic diagram of a process of guiding low-level spatial feature fusion using a priori basis according to an exemplary embodiment;

[0055] Figure 4 1 is a schematic diagram of a process of adaptive fusion of high-level feature channels according to an exemplary embodiment;

[0056] Figure 5 A schematic diagram of a process of generating a defect segmentation map by decoding according to an exemplary embodiment;

[0057] Figure 6 The figure is a structural diagram of an RGB-D rail defect segmentation system based on prior guidance and channel fusion according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] The present application is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present application, but are not intended to limit the present application in any form. It should be noted that, without departing from the concept of the present application, a number of variations and improvements may be made by those skilled in the art, and these all fall within the scope of protection of the present application. Parts not described in detail in the following examples may be implemented using existing technologies.

[0059] Glossary:

[0060] Separate gradient kernel: consists of two one-dimensional convolutions for feature extraction.

[0061] Existing technologies suffer from limitations such as low-contrast texture interference on railway surfaces and blurred defect boundaries, making it difficult for existing technologies to meet the comprehensive requirements of industrial inspection for accuracy, speed, and robustness. To address these issues, the present invention provides an RGB-D rail defect segmentation method based on prior guidance and channel fusion to address these issues.

[0062] Reference Figure 1 As shown, in one embodiment of the present application, a RGB-D rail defect segmentation method based on prior guidance and channel fusion includes the following steps:

[0063] S1, obtains the RGB image and depth image of the rail surface;

[0064] S2, extracts multi-scale heterogeneous global features of RGB images and depth images, and divides them into low-level features and high-level features;

[0065] S3, performs prior-guided spatial fusion on low-level features to obtain low-level fusion features;

[0066] S4, performing channel adaptive fusion on high-level features to obtain high-level fusion features;

[0067] S5, decoding the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface.

[0068] Specifically, prior guidance refers to using the characteristics of defect images, such as similar backgrounds, uneven textures, and irregular edges, to perform targeted spatial feature processing.

[0069] Channel adaptation refers to the channel separation, reconstruction, and adaptive gate calibration of features.

[0070] The above-mentioned embodiments of the present application demonstrate high-precision segmentation capabilities: by fusing the rich texture information of RGB images with the geometric features of depth images, combining spatial enhancement of low-level features with channel-adaptive weighting of high-level features, the detection sensitivity for small defects (such as cracks <1mm in width) is significantly improved. For complex background interference (such as rust, oil stains, and uneven lighting), the layered fusion strategy effectively suppresses noise, achieving a significantly better false detection rate than traditional threshold-based segmentation methods in strong interference scenarios.

[0071] The above-mentioned embodiments of the present application are highly environmentally robust: the multimodal data complementarity design enables the entire segmentation method to maintain stable performance in extreme environments. Prior guidance can enhance defect edge response, solving the problem of missed detection of fuzzy defects (such as surface peeling) by traditional methods.

[0072] In order to obtain the RGB image and depth image of the rail surface, in some specific embodiments of the present application, in the above step S1, the RGB image and depth image of the rail surface are synchronously collected by a binocular camera or an RGB-D sensor.

[0073] Exemplarily, the RGB-D sensor is a depth camera that uses structured light or ToF principles, such as Intel RealSense D455, which supports simultaneous output of high-resolution RGB images and depth maps.

[0074] In order to make the acquired image data more suitable for application, in some embodiments, the acquired image is preprocessed to generate data pairs suitable for input. The preprocessing step includes methods such as denoising, alignment, and normalization.

[0075] For example: Spatial alignment: Mapping the depth map to the RGB coordinate system based on calibration parameters ensures an alignment error of less than 0.5 pixels. Dynamic denoising: Adaptive bilateral filtering is applied to the RGB image, while temporal median filtering is applied to the depth map, taking a weighted average of three consecutive frames. Data normalization: Normalizing RGB pixel values ​​and depth values ​​to the range [0, 1]. These preprocessing methods have a latency of less than 5 milliseconds, enabling real-time data stream processing in inspection scenarios at speeds of up to 60 kilometers per hour.

[0076] In order to quickly obtain multimodal features, in some specific embodiments of the present application, the network structure used for extracting heterogeneous global features in the above step S2 includes a dual-stream encoder, an RGB branch, a depth branch, and a convolutional layer. Among them, the RGB branch is a 4-layer feature extraction network built based on RMT (Retentive Networks Meet Vision Transformers). The depth branch is a 5-layer feature extraction network built based on MobileNetV2.

[0077] Specifically, such as Figure 2 As shown, step S2 can be divided into S21-S24, including:

[0078] S21 uses a dual-stream encoder to extract RGB features of RGB images and depth features of depth images respectively.

[0079] S22, input the RGB feature into the RGB branch, calculate through the RGB branch network, and output the multi-scale feature map {f R 1 ,f R 2 ,f R 3 ,f R 4}.

[0080] S23, input the depth feature into the depth branch, calculate through the depth branch, output 5 scale features, select four of the scale feature maps {f D 1 ,f D 2 ,f D 3 ,f D 4}, corresponding to the multi-scale feature map of RBG.

[0081] S24, the two multi-scale feature maps {f R 1 ,f R 2 ,f R 3 ,f R 4} and {f D 1 ,f D 2 ,f D 3 ,f D 4}, unify the feature dimensions through the convolution layer, classify according to the scale, and obtain the low-level features (f R 1 ,f R 2 and f D 1 ,f D 2 ) and high-level features (f R 3 ,f R 4 and f D 3 ,f D 4 ).

[0082] It should be noted that in other embodiments, the number of layers in the RGB branch and the depth branch is not fixed, but can be adjusted according to specific needs. Therefore, the number of scales, the number of low-level features, and the number of components of high-level features can also be adjusted accordingly.

[0083] In the above-mentioned embodiments of this application, the RMT in the RGB branch uses a local-global hybrid attention mechanism to capture long-range texture dependencies on the rail surface, such as the global correlation of continuous scratches, in the RGB modality. The lightweight design of MobileNetV2 in the depth branch adapts to the low-texture characteristics of the depth map, reducing computational redundancy.

[0084] In order to achieve accurate segmentation of tiny defects in complex backgrounds, in some specific implementations of the present application, in step S3, the low-level features are spatially fused with prior guidance to obtain low-level fusion features, such as Figure 3 As shown, steps S31-S34 may be employed.

[0085] Specifically, S31 performs multi-scale background suppression.

[0086] For RGB feature f R 1 / f R 2 , 3×3 group convolution with expansion rates of 1, 3, and 5 are used to extract multi-scale features. After channel splicing, instance normalization (IN) and Gaussian error linear unit (GELU) activation are performed to obtain the activated multi-scale features. Will Through channel splicing, 1×1 convolution and Sigmoid activation function, the first spatial attention mask is generated.

[0087]

[0088] The above-mentioned multi-scale background suppression method can effectively suppress the interference of rail background texture (such as rust and oil stains) on the defect area.

[0089] S32, perform local variance enhancement.

[0090] For RGB feature RGB feature f R 1 / f R 2 The local variance is calculated with a 5×5 sliding window, normalized by the global variance, and activated by Sigmoid to generate the second spatial attention mask.

[0091]

[0092] Among them, (H, W) and (h, w) represent the feature dimension and pixel index respectively, ∈ represents a very small constant to ensure numerical stability, and σ represents the variance.

[0093] The above local variance enhancement can improve the contrast between defective areas and normal areas. For example, the variance of the peeling area is significantly higher than that of the smooth surface.

[0094] S33, performing directional gradient enhancement.

[0095] For RGB feature f R 1 / f R 2, using horizontal and vertical separable gradient kernel G Rx and G Ry Calculate the gradient amplitude of RGB features and the depth feature f D 1 ,f D 2 , using the separable gradient kernel G Dx and G Dy Calculate the gradient magnitude of the depth modality and normalize it by instance 1×1 convolution and Sigmoid activation generate the third space attention mask weights respectively and the fourth spatial attention mask weight

[0096]

[0097] Specifically, a two-dimensional convolution is generally used for feature extraction. Here, the separable gradient kernel is extracted using two one-dimensional convolutions. Considering the irregular direction, the separation into two directions can adapt to the irregularity of the edge.

[0098] The above-mentioned gradient enhancement technology can enhance the defect edge, such as the linear gradient characteristics of the crack.

[0099] S34, feature fusion.

[0100] RGB features and After element-by-element multiplication, the residuals are added, and the deep features are After element-by-element multiplication, the residuals are added to obtain the fused low-level features.

[0101]

[0102] In the above embodiment of the present application, the entire low-level feature prior-guided spatial fusion is designed based on the prior information of the RGB-D image of the rail surface (prior information of background similarity, texture unevenness, and edge irregularity), combined with the targeted enhancement of multimodal low-level features, so that the entire method can still accurately segment tiny defects under complex backgrounds (such as uneven lighting and oil stains).

[0103] In order to improve the semantic differentiation capability in multi-defect scenarios, in some specific implementations of the present application, in the above step S4, high-level features are adaptively fused to obtain high-level fused features, such as Figure 4 As shown, steps S41-S42 may be employed.

[0104] Specifically, S41, grouping refinement.

[0105] The RGB features and depth features are grouped into local refinement subgroup (C / 4 channels) and global semantic subgroup (3C / 4 channels), respectively.

[0106] Depthwise separable convolution is applied to the local subgroup to enhance the details. After cross-modal splicing, channel shuffling and point-wise convolution are performed to generate two sets of enhanced features f_g1 and f_g2.

[0107] S42, channel attention fusion.

[0108] Perform global average pooling on f_g1 and f_g2 respectively, and generate channel attention weights W_c1 and W_c2 through the fully connected layer and Softmax. Figure 4 "Gating Operation" in.

[0109] Multiply the weights by the two sets of enhanced features f_g1 and f_g2 at the channel level and add them together to obtain the fused high-level features.

[0110] The above-mentioned embodiments of this application achieve channel-level grouping refinement, enabling dynamic weighting to enhance semantic differentiation in multi-defect scenarios, for example, enabling simultaneous detection of cracks and spalling. Furthermore, the lightweight design reduces the number of parameters by 35% compared to traditional channel attention modules.

[0111] In some specific implementations of the present application, in step S5, decoding is performed to generate a defect segmentation map, and the decoder used is composed of 6 depth-separable convolution modules.

[0112] Specifically, such as Figure 5 As shown, it represents the fourth level fusion feature of the highest layer After the first re-parameterized depth-wise separable convolution module, feature refinement is performed, followed by upsampling and fusion features with the third level. Splicing to obtain the first splicing feature;

[0113] The first concatenated features are extracted by the second re-parameterized depthwise separable convolution module, and then refined by the third re-parameterized depthwise separable convolution module to obtain the third-level features.

[0114] The third level features are upsampled and fused with the second level features Splicing to obtain a second splicing feature;

[0115] The second concatenated features are extracted by the fourth re-parameterized depthwise separable convolution module, and then refined by the fifth re-parameterized depthwise separable convolution module to obtain the second-level features.

[0116] The second-level features are upsampled by the fifth re-parameterized depth-wise separable convolution module and fused with the first-level features. Splicing to obtain the third splicing feature;

[0117] The third concatenated feature is further refined through the sixth re-parameterized depth-wise separable convolution module to obtain the first-level feature;

[0118] The first-level features are double-interpolated to the size of the original input RGB image, and the defect area is marked in the form of a binary mask.

[0119] It is important to note that during the training phase, a total of six re-parameterized depthwise separable convolutional modules are used. Each depthwise separable convolutional module is specifically composed of: 3×3 depthwise separable convolution, 1×1 depthwise separable convolution, and the original feature residual. During the inference phase, each re-parameterized depthwise separable convolutional module is combined into a 3×3 depthwise separable convolution.

[0120] In the above embodiment, the trained network structure is compressed during testing. For example, the sum of the results of three parallel convolutions (with the same kernel size) is equivalent to the result of a single convolution using the summed kernel. Therefore, three convolutions can be used during training to improve the model's learning ability, but during test deployment, they can be losslessly compressed into a single convolution, reducing the number of parameters and computational complexity. This achieves a balance between training stability and inference efficiency.

[0121] This application can be processed efficiently in real time. It adopts a lightweight dual-stream encoder (RMT+MobileNetV2) and a heavily parameterized decoder design. The number of parameters of the rail defect segmentation model corresponding to the rail defect segmentation method is only 16.7M. It achieves a real-time inference speed of 45FPS on the NVIDIA Jetson AGX Xavier edge computing platform, and the end-to-end processing delay is less than 50ms, which fully meets the real-time requirements of high-speed mobile scenarios of railway inspection vehicles.

[0122] In order to make the rail defect segmentation model corresponding to the entire RGB-D rail defect segmentation method based on prior guidance and channel fusion more accurate, in some specific implementations of the present application, the model is trained and optimized.

[0123] First, design the loss function. The total loss L_total = L_BCE + λ·L_IoU, where L_BCE is the binary cross entropy loss, L_IoU is the intersection-over-union loss, and λ is the balance coefficient, which defaults to 0.5.

[0124] Next, the parameters are updated. Based on the above total loss, the alternating direction multiplier method (ADMM) is used to optimize the model parameters, alternating between updating the dual-stream encoder weights and decoder parameters until convergence.

[0125] Based on the above embodiment, the rail defect segmentation model is trained based on the following steps:

[0126] First, determine the training sample set, which includes rail surface defect image samples and their corresponding segmentation ground truth images;

[0127] Specifically, the training sample set can be constructed by using 1500 pairs of RGB-D images in the NEU RSDDS-AUG dataset as rail surface defect image samples and their corresponding segmentation truth maps.

[0128] When constructing the training sample set, the rail surface defect image samples and their corresponding segmentation ground truth maps can be expanded by rotation and flipping, and their sizes can be adjusted to 3×352×352.

[0129] Then, the rail surface defect image sample is input into the initial segmentation model to obtain the segmentation result output by the initial segmentation model;

[0130] Then, based on the segmentation results and the segmentation ground truth map, the binary cross entropy loss and intersection-over-union loss are calculated, and the total loss is calculated based on the binary cross entropy loss and intersection-over-union loss;

[0131] Finally, based on the total loss, the gradient descent method is used to iteratively update the structural parameters of the initial segmentation model to obtain the rail defect segmentation model.

[0132] Specifically, using the total loss and alternating direction multiplier descent method, the initial learning rate is set to 10-4, the learning rate decay cycle is reduced to 1 / 10 of the original learning rate after every 30 complete iterations of the training sample set (epoch), and the batch size is 16. The structural parameters of the initial segmentation model are iteratively updated until the preset number of iterations is reached or the total loss converges, thereby obtaining the rail defect segmentation model. Here, the preset number of iterations can be 100 epochs.

[0133] Afterwards, a test sample set may be determined. The test sample set may be constructed by using 1500 pairs of RGB-D images in the NEU RSDDS-AUG dataset as rail surface defect image samples and their corresponding segmentation truth maps.

[0134] During testing, the rail surface defect RGB-D image samples were resized to 3×352×352 and then fed into the rail defect segmentation model to obtain the segmentation results. The segmentation results were then resized to the same size as the input rail surface defect image samples.

[0135] Thereafter, the segmentation results obtained during the test process can be used to calculate the segmentation truth map corresponding to the rail surface defect image sample, and the test index can be used to determine the performance of the rail defect segmentation model.

[0136] The above-mentioned embodiments of this application have significant industrial practical value. End-to-end integration supports dynamic data collection and real-time analysis. After replacing manual visual inspections, the cost of a single inspection is reduced by 60%, and the defect missed detection rate is less than 0.1%. This can avoid major accidents caused by missed defects and economic losses exceeding 10 million yuan each year. The output results can be directly connected to the railway operation and maintenance system to support defect lifecycle management (such as deterioration trend prediction), helping to extend the maintenance cycle from 3 months to 6 months, and increasing the service life of the rails by more than 30%.

[0137] This application is also generalizable and scalable. Its modular design allows for flexible adaptation to different sensors (e.g., ToF / binocular / structured light cameras) and hardware platforms (e.g., embedded devices / cloud servers). With just a few sample adjustments, it can be migrated to other industrial defect detection scenarios (e.g., bridge cracks, pipeline corrosion), demonstrating broad applicability.

[0138] Based on the same technical concept, an embodiment of the present application provides an RGB-D rail defect segmentation system based on prior guidance and channel fusion, such as Figure 6 As shown, including:

[0139] Image acquisition module, which acquires RGB images and depth images of the rail surface;

[0140] The global heterogeneous extraction module extracts multi-scale heterogeneous global features of RGB images and depth images, dividing them into low-level features and high-level features;

[0141] The prior guidance module performs prior-guided spatial fusion on low-level features to obtain low-level fusion features;

[0142] The channel adaptive module performs channel adaptive fusion on high-level features to obtain high-level fusion features;

[0143] The defect segmentation module decodes the low-level fusion features and the high-level fusion features to obtain the defect segmentation map of the rail surface.

[0144] During implementation, the result display module transmits the segmentation visualization results to the operation and maintenance system. For example, defect areas are overlaid on the original RGB image as a red semi-transparent mask, with edges highlighted in yellow. The system displays defect type (e.g., cracks, spalling, pits), area percentage, and location coordinates in real time.

[0145] In terms of data interface, the system outputs data in JSON format through ROS topics or HTTP API, including:

[0146]

[0147] In addition, the system supports integration with PLC systems via the Modbus / TCP protocol to trigger real-time alarm signals. The output delay of the embodiment of the present application is extremely low, which can meet the real-time response requirements of the railway inspection vehicle control system.

[0148] The specific implementation techniques of each module / unit in the above example of this application can refer to the corresponding steps of the RGB-D rail defect segmentation method based on prior guidance and channel fusion in the above embodiment, which will not be repeated here.

[0149] The preferred features of the above embodiments can be used alone in any embodiment, or in any combination without conflict. In addition, parts not described in detail in the embodiments can be implemented using existing technologies.

[0150] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDRSDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc. The above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0151] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0152] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0153] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0154] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0155] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0158] The above describes some specific embodiments of the present application. It should be understood that the present application is not limited to the specific embodiments described above, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the substantive content of the present application. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A RGB-D rail defect segmentation method based on prior guidance and channel fusion, characterized in that: include: Obtain RGB images and depth images of the rail surface; Extracting multi-scale heterogeneous global features of the RGB image and the depth image, and dividing them into low-level features and high-level features; Performing a priori-guided spatial fusion on the low-level features to obtain low-level fused features; Performing channel adaptive fusion on the high-level features to obtain high-level fused features; A decoding operation is performed on the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface.

2. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 1 is characterized in that: The RGB-D rail defect segmentation model is used to extract multi-scale heterogeneous global features of the RGB image and depth image, which are divided into low-level features and high-level features, including: A dual-stream encoder is used to extract RGB features and depth features respectively; The RGB features are fed into the RGB branch, and multiple scale RGB feature maps are output through a multi-layer feature extraction network constructed by RMT; The depth features are fed into the depth branch, and multiple scale depth feature maps are output through a multi-layer feature extraction network constructed by MobileNetV2; A convolutional layer is used to unify the feature dimensions of the multiple scale RGB feature maps and the multiple scale depth feature maps, and group them into low-level features and high-level features according to their scale sizes.

3. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 1 is characterized in that: The prior-guided spatial fusion of the low-level features to obtain low-level fusion features includes: Background suppression is performed on the RGB features in the low-level features, and multi-scale features are extracted using group convolution with different dilation rates. The multi-scale features are activated by instance normalization and Gaussian error linear unit respectively, and the activated multi-scale features are channel-joined. The first spatial attention mask weight is generated through a convolution layer and Sigmoid activation; For the RGB features in the low-level features, the local variance is calculated using a fixed-size sliding window, and the second spatial attention mask weight is generated through variance normalization and Sigmoid activation; Calculating the gradient magnitudes of the RGB features and the depth features using horizontal and vertical separable gradients, and generating the third spatial attention mask weight and the fourth spatial attention mask weight respectively through strength normalization, convolution, and Sigmoid activation; The RGB features in the low-level features are element-wise multiplied with the first, second, and third spatial attention mask weights respectively, and the depth features are element-wise multiplied with the fourth spatial attention mask weights, and then residuals are added with the RGB features and the depth features to obtain the enhanced features after fusion at this level, i.e., the low-level fusion features.

4. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 1 is characterized in that: The performing channel adaptive fusion on the high-level features to obtain high-level fusion features includes: Grouping the RGB features and the depth features of the high-level features into local refinement subgroups and global semantic subgroups respectively; Applying depthwise separable convolution to the local refined subgroups of RGB features and depth features to enhance details; For the global semantic subgroup and the local refined subgroup after detail enhancement, cross-channel splicing of two sets of cross-modal features is performed to obtain two sets of fused convolutions; Shuffling the channels of the two groups of fused convolutions and performing point-by-point convolution to obtain a first group of enhanced features and a second group of enhanced features; Perform global average pooling and two fully connected layers and Softamax on the first set of enhanced features and the second set of enhanced features to obtain channel attention weights; The channel attention weights are respectively multiplied by the first set of enhanced features and the second set of enhanced features at the channel level, and the enhanced features are element-wise added to finally obtain high-level fused features, i.e., high-level fused features.

5. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 1 is characterized in that: The decoding operation on the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface includes: Arrange the low-level and high-level fusion features in descending order of scale; The highest-level fusion features are refined through a re-parameterized depth-wise separable convolution module; The refined features are upsampled and concatenated with the previous level features to obtain new level features; Repeat the above process twice to obtain a feature map of the same size as the first-level feature; The feature map obtained above is interpolated and upsampled to obtain the final defect segmentation map.

6. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 5 is characterized in that: During the training phase, each of the re-parameterized depth-wise separable convolutions includes three summation parts: a 3×3 depth-wise separable convolution, a 1×1 depth-wise separable convolution, and the original feature residual. During the inference phase: each of the re-parameterized depth-wise separable convolutions is a 3×3 depth-wise separable convolution.

7. The RGB-D rail defect segmentation method based on prior guidance and channel fusion according to claim 2 is characterized in that: The training process of the rail defect segmentation model corresponding to the segmentation method includes: Determine a training sample set, including samples of rail surface defect RGB images and depth images and their corresponding segmentation ground truth maps; Inputting the samples into the initial segmentation model respectively to obtain the segmentation results output by the initial segmentation model; Calculating a binary cross entropy loss and an intersection-over-union loss based on the segmentation result and the segmentation truth map, and calculating a total loss based on the binary cross entropy loss and the intersection-over-union loss; Based on the total loss, the alternating direction multiplier method is used to iteratively update the structural parameters of the initial segmentation model to obtain the rail defect segmentation model.

8. An RGB-D rail defect segmentation system based on prior guidance and channel fusion, characterized by: include: Image acquisition module, which acquires RGB images and depth images of the rail surface; A global heterogeneous extraction module extracts multi-scale heterogeneous global features of the RGB image and the depth image and divides them into low-level features and high-level features; A priori guided module performs a priori guided spatial fusion on the low-level features to obtain low-level fusion features; A channel adaptive module performs channel adaptive fusion on the high-level features to obtain high-level fusion features; The defect segmentation module decodes the low-level fusion features and the high-level fusion features to obtain a defect segmentation map of the rail surface.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 7, or run the system according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 7, or to run the system according to claim 8.

Images