Railway steel rail surface state detection method and system based on image semantic segmentation

Abstract

The invention provides a railway steel rail surface state detection method and system based on image semantic segmentation, and relates to the technical field of railway steel rail detection, an image data set of a steel rail surface state comprises a light band image, a fish scale pattern image, a chipping image and semantic segmentation labels corresponding to various images; performing feature extraction processing according to the image data set, and respectively extracting shallow semantic information and deep semantic information based on the complexity of the surface state of the steel rail to obtain a preliminary feature map; performing feature fusion processing on the preliminary feature map to obtain a multi-scale feature fusion result; and constructing a railway steel rail surface state detection model based on the multi-scale feature fusion result, and detecting the surface of the steel rail to be detected by using the railway steel rail surface state detection model to obtain a classification result of the actual steel rail surface state. The problem that pixel-level detection cannot be carried out on light bands, falling blocks and fish scale lines on the surface of the railway steel rail at the same time in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of railway rail detection, and in particular to a railway rail surface state detection method and system based on image semantic segmentation. Background Art

[0002] The accumulation of track defects not only affects the smooth operation of trains, but may also cause transportation delays and accidents. Therefore, regular inspection and identification of defects such as light bands, dropped blocks, and fish scales can detect track problems at an early stage, prevent problems from accumulating to a serious level, and reduce emergency repair work due to sudden failures.

[0003] In the existing technology, the detection of light bands, fallen blocks and fish scale patterns on rails in my country still mainly relies on manual work. The automatic detection methods for fallen blocks and fish scale patterns usually use target detection methods and traditional machine vision methods, and do not involve defect detection with pixel-level accuracy. Although the automatic detection method for light bands has pixel-level accuracy, it cannot detect surface damage at the same time.

[0004] In addition, the existing automatic detection methods for rail block and fish scale patterns based on machine vision cannot achieve high-precision pixel-level detection, have high development costs, poor robustness, and difficulty in handling complex scenes. The existing light band automatic detection methods cannot detect both block and fish scale patterns at the same time. The detection of block and fish scale patterns requires the development of new algorithms, and the application scenarios are limited. At the same time, the existing pixel-level precision detection technology cannot detect multiple categories of surface conditions at the same time. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for detecting the surface state of railway rails based on image semantic segmentation to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows: In a first aspect, the present application provides a method for detecting the surface state of a railway rail based on image semantic segmentation, comprising: Acquire an image dataset of rail surface conditions, wherein the image dataset of rail surface conditions includes light band images, fish scale pattern images, block drop images, and semantic segmentation labels corresponding to various images; Performing feature extraction processing according to the image data set, respectively extracting shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; Performing feature fusion processing on the preliminary feature map, obtaining a multi-scale feature fusion result by combining shallow semantic information and deep semantic information and restoring image resolution; A railway rail surface state detection model is constructed based on the multi-scale feature fusion result, and the railway rail surface state detection model is used to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

[0006] In a second aspect, the present application also provides a railway rail surface condition detection system based on image semantic segmentation, comprising: An acquisition subsystem is used to acquire an image dataset of the surface state of a rail, wherein the image dataset of the surface state of a rail includes a light band image, a fish scale pattern image, a block drop image, and semantic segmentation labels corresponding to each type of image; A feature extraction subsystem is used to perform feature extraction processing according to the image data set, and extract shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; A feature fusion subsystem is used to perform feature fusion processing on the preliminary feature map, and obtain a multi-scale feature fusion result by combining shallow semantic information and deep semantic information and restoring image resolution; The detection subsystem is used to construct a railway rail surface state detection model based on the multi-scale feature fusion result, and use the railway rail surface state detection model to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

[0007] The beneficial effects of the present invention are: The present invention can efficiently fuse deep semantic information and shallow semantic information by introducing a double-layer feature fusion module, and extract the features of light bands, dropped blocks and fish scales with high precision. At the same time, the double-layer feature fusion module is embedded to obtain a railway rail surface state detection model. The structure of the railway rail surface state detection model with unchanged resolution ensures the detection accuracy of multi-classification semantic segmentation, and combined with the double-layer feature fusion module, the detection performance is further improved, thereby optimizing the overall recognition effect of the model.

[0008] Other features and advantages of the present invention will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 It is a schematic flow chart of a method for detecting the surface state of a railway rail based on image semantic segmentation according to an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a double-layer feature fusion module in an embodiment of the present invention; Figure 3It is a structural schematic diagram of a railway rail surface state detection model including three codecs in an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of a railway rail surface condition detection system based on image semantic segmentation described in an embodiment of the present invention.

[0011] Markings in the figure: 901, acquisition subsystem; 902, feature extraction subsystem; 903, feature fusion subsystem; 904, detection subsystem; 9021, first construction unit; 9022, weighting unit; 9023, first extraction unit; 9024, second extraction unit; 9025, output unit; 9031, second construction unit; 9032, feature splicing unit; 9033, adjustment unit; 9041, third construction unit; 9042, fourth construction unit; 9043, definition unit; 9044, fifth construction unit. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0013] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0014] Embodiment 1: This embodiment provides a method for detecting the surface state of railway rails based on image semantic segmentation.

[0015] See also Figure 1 , the figure shows that the method includes step S100, step S200, step S300, and step S400.

[0016] Step S100: Acquire an image dataset of the surface state of a rail, wherein the image dataset of the surface state of a rail includes a light band image, a fish scale pattern image, a block drop image, and semantic segmentation labels corresponding to each type of image; In this embodiment, a hand-push track inspection cart integrated with a 3D camera and a 2D camera is used to collect image data of different types of light bands, broken blocks and fish scales on the rail surface, and each image is annotated with light bands, broken blocks and fish scales to generate corresponding semantic segmentation labels to obtain an image data set. The image data set is then divided into a training set and a validation set in a certain proportion for model training and evaluation.

[0017] Step S200: performing feature extraction processing according to the image data set, extracting shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; In this embodiment, the shallow semantic information contained in the shallow semantic features is related to the basic visual features in the image, such as edges, textures, colors, and simple geometric shapes, while the deep semantic information contained in the deep semantic features has more contextual information and overall information, such as the shape, posture, structure, and semantic understanding of the object. Therefore, the abstraction and combination of multi-layer networks are required to extract the complete deep semantic information.

[0018] Specifically, in the multi-classification recognition of light bands, dropouts and fish scales, light bands are easier to identify in multi-classification tasks because of their regular shape and obvious features, which are less dependent on the depth of the detection model. Fish scales are typical shallow semantic features, and their semantic information is easily lost as the depth of the detection model increases. Dropouts are deep semantic features, and their semantic information is strengthened as the depth of the detection model increases.

[0019] Simultaneous identification of light bands, dropped blocks, and fish scales places high demands on the detection model. Otherwise, shallow semantic information is likely to lose its significance when the depth of the detection model increases, resulting in information loss and easy misidentification of fish scales. Therefore, based on the differences in the characteristics of the three targets, this embodiment designs a double-layer feature fusion module that efficiently fuses deep semantic information and layer semantic information, wherein the double-layer feature fusion module includes a convolutional attention mechanism submodule, a shallow semantic extraction submodule, a deep semantic extraction submodule, and a feature fusion submodule. The double-layer feature fusion module can simultaneously focus on the shallow semantic information and deep semantic information of the image.

[0020] The step S200 includes: Step S201: According to the complexity of the surface state of the rail in the image data set, a convolutional attention mechanism submodule, a shallow semantic extraction submodule and a deep semantic extraction submodule are constructed, wherein the shallow semantic extraction submodule includes a hole convolution layer, a depth-separable convolution layer, a maximum pooling layer, a first upsampling layer, a first convolution layer, a first batch of normalization and linear rectification layers, a second batch of normalization and linear rectification layers, and a third batch of normalization and linear rectification layers, and the deep semantic extraction submodule includes a second convolution layer, a third convolution layer, an average pooling layer, a second upsampling layer, a fourth batch of normalization and linear rectification layers, and a fifth batch of normalization and linear rectification layers; In this embodiment, Figure 2 The figure shows the structure of the double-layer feature fusion module, where the dilated convolution layer uses a 3×3 convolution kernel with a dilated rate of 6, and the depthwise separable convolution layer uses a 3×3 convolution kernel. At the same time, the first and third convolution layers use 1×1 convolution kernels, the second convolution layer uses a 3×3 convolution kernel, and the first batch normalization and linear rectification layer, the second batch normalization and linear rectification layer, the third batch normalization and linear rectification layer, the fourth batch normalization and linear rectification layer, the fifth batch normalization and linear rectification layer, and the sixth batch normalization and linear rectification layer are all composed of a batch normalization and a ReLU activation function.

[0021] Step S202: performing a weighted operation on the input feature map of the convolutional attention mechanism submodule to obtain a first output feature map; In this embodiment, the convolutional attention mechanism submodule includes a convolutional attention mechanism CBAM, which weights the input feature map through the convolutional attention mechanism CBAM to enhance the attention to key features, wherein the calculation formula of the first output feature map is: ; ; ; ; In the formula, represents the first output feature map, represents the convolutional attention mechanism CBAM, represents the input feature map of the convolutional attention mechanism submodule, represents the output feature map of the spatial attention module, represents the output feature map of the channel attention module, represents the sigmoid function, represents matrix multiplication, and Represents the average pooling feature map and the maximum pooling feature map of the entire channel respectively and denote the first weight parameter and the second weight parameter respectively, and Represent the average pooling feature map and the maximum pooling feature map respectively, Represents a 3×3 convolution kernel.

[0022] Step S203: extracting shallow semantic information of the first output feature map through the shallow semantic extraction submodule to obtain a second output feature map; The step S203 includes: Step A100: extracting shallow semantic information of the first output feature map through a dilated convolution layer and a depthwise separable convolution layer respectively to obtain a first shallow feature and a second shallow feature; Step A200: performing batch normalization and linear rectification operations on the first shallow feature in sequence through a first batch normalization and linear rectification layer to obtain a first activation feature; Step A300: performing batch normalization and linear rectification operations on the second shallow feature in turn through a second batch normalization and linear rectification layer to obtain a second activation feature; Step A400: performing feature splicing on the first activation feature and the second activation feature to obtain a first splicing feature map; Step A500: The first spliced ​​feature map is sequentially passed through a maximum pooling layer and a first upsampling layer to restore the resolution of the first spliced ​​feature map to obtain a restored feature map; Step A600: The restored feature map is sequentially passed through the first convolution layer and the third batch normalization and linear rectification layer to adjust the number of channels to obtain a second output feature map.

[0023] In this embodiment, shallow semantic information is extracted by dilated convolution and depthwise separable convolution respectively, and after batch normalization and ReLU activation function activation, splicing is performed along a specific dimension to obtain a first spliced ​​feature map, and then the semantic features are extracted by the maximum pooling operation and then up-sampled to restore the resolution. Finally, the number of channels is adjusted by 1×1 convolution kernel, batch normalization and ReLU activation function to obtain the second output feature map. Therefore, the shallow semantic extraction submodule focuses on the shallow semantic features of the input feature map, where the calculation formula of the second output feature map is: ; ; ; ; In the formula, represents the second output feature map, represents a 1×1 convolution kernel, represents the upsampling operation, represents the maximum pooling operation, represents the first activated feature, represents the second activation feature, represents the first concatenated feature map, represents a dilation rate of 6 and a dilated convolution of size 3×3. represents the first output feature map, represents a 3×3 depth-wise separable convolution, represents the feature concatenation operation, represents the ReLU activation function, Represents batch normalization.

[0024] Step S204: extracting deep semantic information of the first output feature map through the deep semantic extraction submodule to obtain a third output feature map; The step S204 includes: Step B100: extracting features from the first output feature map through a second convolutional layer to obtain first extracted features; Step B200: performing batch normalization and linear rectification operations on the first extracted features in sequence through a fourth batch normalization and linear rectification layer to obtain a first deep feature; Step B300: performing average pooling and upsampling operations on the first output feature map in turn through an average pooling layer and a second upsampling layer to obtain a second deep feature; Step B400: performing feature splicing on the first deep feature and the second deep feature to obtain a second splicing feature map; Step B500: Pass the second concatenated feature map through the third convolution layer and the fifth batch normalization and linear rectification layer in sequence to adjust the number of channels to obtain a third output feature map.

[0025] In this embodiment, a 3×3 convolution kernel, batch normalization and ReLU activation function, as well as an average pooling operation and an upsampling operation are used to extract the deep semantic information of the input image, and then the number of channels is adjusted through a 1×1 convolution kernel, batch normalization and ReLU activation function to obtain a third output feature map with deep semantic information.

[0026] Among them, the calculation formula of the third output feature map is: ; ; ; ; In the formula, represents the third output feature map, represents the ReLU activation function, represents batch normalization, represents the feature concatenation operation, represents a 1×1 convolution kernel, represents the first output feature map, represents a 3×3 convolution kernel, represents the upsampling operation, represents the average pooling operation, represents the first deep feature, represents the second deep feature, Represents the second concatenated feature map.

[0027] Step S205: taking the second output feature map and the third output feature map together as a preliminary feature map.

[0028] Step S300: performing feature fusion processing on the preliminary feature map, combining shallow semantic information and deep semantic information and restoring image resolution to obtain a multi-scale feature fusion result; The step S300 includes: Step S301: constructing a feature fusion submodule, wherein the feature fusion submodule includes a fourth convolutional layer and a sixth batch normalization and linear rectification layer; Step S302: combining shallow semantic information and deep semantic information to perform feature splicing on the second output feature map and the third output feature map to obtain a third spliced ​​feature map; Step S303: The third concatenated feature map is sequentially passed through the fourth convolution layer and the sixth batch normalization and linear rectification layer to adjust the number of channels, restore the image resolution, and obtain a multi-scale feature fusion result.

[0029] In this embodiment, the fourth convolution layer in the feature fusion submodule adopts a 1×1 convolution kernel. At the same time, the feature fusion submodule splices the features extracted by the shallow semantic extraction submodule and the deep semantic extraction submodule, adjusts the number of channels, and obtains a multi-scale feature fusion result that is consistent with the size of the input feature map. The calculation formula of the multi-scale feature fusion result is: ; In the formula, represents the multi-scale feature fusion result, represents a 1×1 convolution kernel, represents the ReLU activation function, Represents batch normalization.

[0030] Step S400: constructing a railway rail surface state detection model based on the multi-scale feature fusion result, and using the railway rail surface state detection model to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

[0031] The step S400 includes: Step S401: constructing a ShuttleNet model, wherein the ShuttleNet model includes a plurality of sequentially connected codecs, wherein the codec includes a coding layer and a decoding layer with a plurality of resolution levels, wherein the plurality of resolution levels of each coding layer are connected by a downsampling operation, and the plurality of decoding layers of each resolution level are connected by an upsampling operation; Step S402: constructing a double-layer feature fusion module based on the convolutional attention mechanism submodule, the shallow semantic extraction submodule, the deep semantic extraction submodule and the feature fusion submodule; Step S403: using the multi-scale feature fusion result output by the feature fusion submodule as the output feature of the double-layer feature fusion module; Step S404: The downsampling unit of each resolution level and the output end of the encoding layer of the first four resolution levels in the ShuttleNet model are embedded in the double-layer feature fusion module to obtain a railway rail surface state detection model.

[0032] In this embodiment, the detection of light bands, chipping and fish scales is actually a multi-classification recognition task. In multi-classification recognition tasks, how to fuse features at different levels is the key difficulty in building a high-precision semantic segmentation model. The ShuttleNet model is a shuttle semantic segmentation model, which uses 3×3 convolution kernels, batch normalization, ReLU activation functions and memory connections at the same resolution level to form a series of feature representations with unchanged spatial size. The ShuttleNet model enables high-precision recognition of light bands. However, the shapes of chipping and fish scales are irregular, and there are interferences such as grinding marks and stains on the rails, which increase the difficulty of identifying chipping and fish scales. At the same time, as the depth of the ShuttleNet model increases, the characteristic information of fish scales is gradually lost.

[0033] Therefore, in order to further improve the recognition accuracy of light bands, dropped blocks and fish scale patterns, a double-layer feature fusion module is embedded in the output end of the downsampling unit of each resolution level and the encoding layer of the first four resolution levels in the ShuttleNet model to obtain a pixel-level railway rail surface status detection model.

[0034] The structure of the railway rail surface state detection model can not only effectively extract the deep semantic information of the image, but also retain the shallow semantic information at the same time, balance the contradiction between blocking and fish scale patterns in the recognition process, and ensure high-precision pixel-level recognition of light bands. Among them, the number of encoders and decoders of the railway rail surface state detection model can have different settings, which can be selected according to the actual application.

[0035] like Figure 3 As shown in the figure, it is a structural diagram of the railway rail surface state detection model containing three encoders and decoders, where Average Pooling represents average pooling, 3×3Conv represents 3×3 convolution kernel, Softmax represents normalized exponential function, BN represents batch normalization, ReLU represents ReLU activation function, concatenation represents concatenation operation, 3×3DeConv represents 3×3 deconvolution kernel, 2×2 Max Pooling represents 2×2 maximum pooling, memory connection represents memory connection, and output represents the output result of the railway rail surface state detection model. Figure 3 The 4, 16, 32, 64, 128, and 256 in the above table represent the number of feature map channels, i.e., the depth.

[0036] like Figure 3 As shown, the three codecs are respectively the first codec, the second codec and the third codec, and include five resolution levels, namely the first resolution level, the second resolution level, the third resolution level, the fourth resolution level and the fifth resolution level, wherein the height and width of the feature map of the first resolution level are H×W, and the height and width of the feature maps of the second resolution level, the third resolution level, the fourth resolution level and the fifth resolution level are respectively , , and .

[0037] The different resolution levels in each encoding layer of the railway rail surface state detection model are connected through downsampling operations, which include sequentially connected 2×2 maximum pooling, double-layer feature fusion modules, and 2×2 maximum pooling. At the same time, the different resolution levels in each decoding layer of the railway rail surface state detection model are connected through upsampling operations, which include sequentially connected 3×3 deconvolution kernels, batch normalization, and ReLU activation functions.

[0038] In this embodiment, the double-layer feature fusion module can avoid the loss of shallow semantic information as the depth of the railway rail surface state detection model increases. The unique structure of the double-layer feature fusion module enhances the multi-scale feature fusion capability of the railway rail surface state detection model, and can effectively combine deep and shallow semantic features to improve the recognition performance of the railway rail surface state detection model.

[0039] In this embodiment, when using the railway rail surface state detection model to detect the surface of the rail to be detected, it also includes training and verifying the railway rail surface state detection model through an image data set to obtain a trained railway rail surface state detection model, and detecting the surface of the rail to be detected through the trained railway rail surface state detection model to obtain a classification result of the actual rail surface state.

[0040] Among them, when training and verifying, a data set containing 2610 annotated images was selected, including 1800 images as training sets and 810 images as verification sets. The railway rail surface state detection model was trained for 30 rounds using the training set, and verified using the verification set. The results showed that compared with the milestone model Deeplabv3+, the railway rail surface state detection model proposed in the present invention has an average intersection-over-union ratio increased by 2.33%, and a macro F1 value increased by 0.3%, but the number of parameters is only 1 / 4 of the milestone model Deeplabv3+. Therefore, the railway rail surface state detection model based on the double-layer feature fusion module of the present invention has the characteristics of lightweight, and can automatically learn the features of light bands, dropped blocks and fish scales, handle complex scenes, and achieve high-precision pixel-level detection, which solves the problem that the existing technology cannot simultaneously achieve pixel-level detection of light bands, dropped blocks and fish scales.

[0041] Embodiment 2: like Figure 4 As shown, this embodiment provides a railway rail surface state detection system based on image semantic segmentation, and the system includes: An acquisition subsystem 901 is used to acquire an image dataset of the surface state of a rail, wherein the image dataset of the surface state of a rail includes a light band image, a fish scale pattern image, a block drop image, and semantic segmentation labels corresponding to each type of image; A feature extraction subsystem 902 is used to perform feature extraction processing according to the image data set, and extract shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; The feature fusion subsystem 903 is used to perform feature fusion processing on the preliminary feature map, and obtain a multi-scale feature fusion result by combining shallow semantic information and deep semantic information and restoring image resolution; The detection subsystem 904 is used to construct a railway rail surface state detection model based on the multi-scale feature fusion result, and use the railway rail surface state detection model to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

[0042] The feature extraction subsystem 902 includes: A first construction unit 9021 is used to construct a convolutional attention mechanism submodule, a shallow semantic extraction submodule and a deep semantic extraction submodule according to the complexity of the surface state of the rail in the image data set, wherein the shallow semantic extraction submodule includes a hole convolution layer, a depthwise separable convolution layer, a maximum pooling layer, a first upsampling layer, a first convolution layer, a first batch of normalization and linear rectification layers, a second batch of normalization and linear rectification layers, and a third batch of normalization and linear rectification layers, and the deep semantic extraction submodule includes a second convolution layer, a third convolution layer, an average pooling layer, a second upsampling layer, a fourth batch of normalization and linear rectification layers, and a fifth batch of normalization and linear rectification layers; A weighting unit 9022 is used to perform a weighted operation on the input feature map of the convolutional attention mechanism submodule to obtain a first output feature map; A first extraction unit 9023 is used to extract shallow semantic information of the first output feature map through the shallow semantic extraction submodule to obtain a second output feature map; A second extraction unit 9024 is used to extract deep semantic information of the first output feature map through the deep semantic extraction submodule to obtain a third output feature map; The output unit 9025 is used to use the second output feature map and the third output feature map together as a preliminary feature map.

[0043] The feature fusion subsystem 903 includes: A second construction unit 9031 is used to construct a feature fusion submodule, wherein the feature fusion submodule includes a fourth convolutional layer and a sixth batch normalization and linear rectification layer; A feature concatenation unit 9032 is used to concatenate the second output feature map and the third output feature map by combining shallow semantic information and deep semantic information to obtain a third concatenated feature map; The adjustment unit 9033 is used to adjust the number of channels of the third spliced ​​feature map through the fourth convolution layer and the sixth batch normalization and linear rectification layer in sequence, restore the image resolution, and obtain a multi-scale feature fusion result.

[0044] The detection subsystem 904 includes: The third construction unit 9041 is used to construct a ShuttleNet model, wherein the ShuttleNet model includes a plurality of codecs connected in sequence, wherein the codec includes a coding layer and a decoding layer with a plurality of resolution levels, wherein the plurality of resolution levels of each coding layer are connected by a downsampling operation, and the plurality of decoding layers of each resolution level are connected by an upsampling operation; A fourth construction unit 9042 is used to construct a double-layer feature fusion module based on the convolutional attention mechanism submodule, the shallow semantic extraction submodule, the deep semantic extraction submodule and the feature fusion submodule; A definition unit 9043 is used to use the multi-scale feature fusion result output by the feature fusion submodule as the output feature of the double-layer feature fusion module; The fifth construction unit 9044 is used to embed the double-layer feature fusion module into the output ends of the downsampling operation of each resolution level and the encoding layers of the first four resolution levels in the ShuttleNet model to obtain a railway rail surface state detection model.

[0045] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0047] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A railway rail surface state detection method based on image semantic segmentation, characterized in that: include: Acquire an image dataset of rail surface conditions, wherein the image dataset of rail surface conditions includes light band images, fish scale pattern images, block drop images, and semantic segmentation labels corresponding to various images; Performing feature extraction processing according to the image data set, respectively extracting shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; Performing feature fusion processing on the preliminary feature map, obtaining a multi-scale feature fusion result by combining shallow semantic information and deep semantic information and restoring image resolution; A railway rail surface state detection model is constructed based on the multi-scale feature fusion result, and the railway rail surface state detection model is used to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

2. The railway rail surface state detection method based on image semantic segmentation according to claim 1 is characterized in that: Feature extraction processing is performed according to the image data set, and shallow semantic information and deep semantic information are extracted based on the complexity of the rail surface state to obtain a preliminary feature map, including: According to the complexity of the surface state of the rail in the image data set, a convolutional attention mechanism submodule, a shallow semantic extraction submodule and a deep semantic extraction submodule are constructed, wherein the shallow semantic extraction submodule includes a hole convolution layer, a depth-separable convolution layer, a maximum pooling layer, a first upsampling layer, a first convolution layer, a first batch of normalization and linear rectification layers, a second batch of normalization and linear rectification layers, and a third batch of normalization and linear rectification layers, and the deep semantic extraction submodule includes a second convolution layer, a third convolution layer, an average pooling layer, a second upsampling layer, a fourth batch of normalization and linear rectification layers, and a fifth batch of normalization and linear rectification layers; Perform a weighted operation on the input feature map of the convolutional attention mechanism submodule to obtain the first output feature map; Extracting shallow semantic information of the first output feature map through the shallow semantic extraction submodule to obtain a second output feature map; Extracting deep semantic information of the first output feature map through the deep semantic extraction submodule to obtain a third output feature map; The second output feature map and the third output feature map are used together as a preliminary feature map.

3. The railway rail surface condition detection method based on image semantic segmentation according to claim 2 is characterized in that: Extracting shallow semantic information of the first output feature map through the shallow semantic extraction submodule to obtain a second output feature map includes: Extracting shallow semantic information of the first output feature map through a dilated convolution layer and a depthwise separable convolution layer respectively to obtain a first shallow feature and a second shallow feature; The first shallow feature is sequentially subjected to batch normalization and linear rectification operations by a first batch normalization layer and a linear rectification layer to obtain a first activation feature; The second shallow feature is sequentially subjected to batch normalization and linear rectification operations by a second batch normalization and linear rectification layer to obtain a second activation feature; Perform feature splicing on the first activation feature and the second activation feature to obtain a first splicing feature map; Passing the first spliced ​​feature map through a maximum pooling layer and a first upsampling layer in sequence to restore the resolution of the first spliced ​​feature map to obtain a restored feature map; The restored feature map is sequentially passed through the first convolution layer and the third batch normalization and linear rectification layer to adjust the number of channels to obtain a second output feature map.

4. The method for detecting the surface state of railway rails based on image semantic segmentation according to claim 2, characterized in that: The deep semantic information of the first output feature map is extracted by the deep semantic extraction submodule to obtain a third output feature map, including: Performing feature extraction on the first output feature map through a second convolutional layer to obtain a first extracted feature; The first extracted features are subjected to batch normalization and linear rectification operations in sequence through a fourth batch normalization and linear rectification layer to obtain a first deep feature; The first output feature map is sequentially passed through an average pooling layer and a second upsampling layer to perform an average pooling operation and an upsampling operation to obtain a second deep feature; Performing feature splicing on the first deep feature and the second deep feature to obtain a second splicing feature map; The second concatenated feature map is sequentially passed through the third convolution layer and the fifth batch normalization and linear rectification layer to adjust the number of channels to obtain the third output feature map.

5. The method for detecting the surface state of railway rails based on image semantic segmentation according to claim 2, characterized in that: The preliminary feature map is subjected to feature fusion processing, and a multi-scale feature fusion result is obtained by combining shallow semantic information and deep semantic information and restoring image resolution, including: Constructing a feature fusion submodule, wherein the feature fusion submodule includes a fourth convolutional layer and a sixth batch normalization and linear rectification layer; Combining shallow semantic information and deep semantic information, the second output feature map and the third output feature map are feature spliced ​​to obtain a third spliced ​​feature map; The third spliced ​​feature map is sequentially passed through the fourth convolution layer and the sixth batch normalization and linear rectification layer to adjust the number of channels, restore the image resolution, and obtain a multi-scale feature fusion result.

6. The method for detecting the surface state of railway rails based on image semantic segmentation according to claim 5, characterized in that: A railway rail surface state detection model is constructed based on the multi-scale feature fusion result, including: Constructing a ShuttleNet model, the ShuttleNet model comprising a plurality of sequentially connected codecs, the codec comprising a coding layer and a decoding layer with a plurality of resolution levels, wherein the plurality of resolution levels of each coding layer are connected by a downsampling operation, and the plurality of decoding layers of each resolution level are connected by an upsampling operation; A double-layer feature fusion module is constructed based on the convolutional attention mechanism submodule, the shallow semantic extraction submodule, the deep semantic extraction submodule and the feature fusion submodule; The multi-scale feature fusion result output by the feature fusion submodule is used as the output feature of the double-layer feature fusion module; The double-layer feature fusion module is embedded in the output ends of the downsampling units of each resolution level and the encoding layers of the first four resolution levels in the ShuttleNet model to obtain a railway rail surface state detection model.

7. A railway rail surface condition detection system based on image semantic segmentation, characterized in that: include: An acquisition subsystem is used to acquire an image dataset of the surface state of a rail, wherein the image dataset of the surface state of a rail includes a light band image, a fish scale pattern image, a block drop image, and semantic segmentation labels corresponding to each type of image; A feature extraction subsystem is used to perform feature extraction processing according to the image data set, and extract shallow semantic information and deep semantic information based on the complexity of the rail surface state to obtain a preliminary feature map; A feature fusion subsystem is used to perform feature fusion processing on the preliminary feature map, and obtain a multi-scale feature fusion result by combining shallow semantic information and deep semantic information and restoring image resolution; The detection subsystem is used to construct a railway rail surface state detection model based on the multi-scale feature fusion result, and use the railway rail surface state detection model to detect the surface of the rail to be detected to obtain a classification result of the actual rail surface state.

8. The railway rail surface condition detection system based on image semantic segmentation according to claim 7, characterized in that: The feature extraction subsystem comprises: A first construction unit is used to construct a convolutional attention mechanism submodule, a shallow semantic extraction submodule and a deep semantic extraction submodule according to the complexity of the surface state of the rail in the image data set, wherein the shallow semantic extraction submodule includes a hole convolution layer, a depth-separable convolution layer, a maximum pooling layer, a first upsampling layer, a first convolution layer, a first batch of normalization and linear rectification layers, a second batch of normalization and linear rectification layers, and a third batch of normalization and linear rectification layers, and the deep semantic extraction submodule includes a second convolution layer, a third convolution layer, an average pooling layer, a second upsampling layer, a fourth batch of normalization and linear rectification layers, and a fifth batch of normalization and linear rectification layers; A weighting unit, used to perform a weighted operation on the input feature map of the convolutional attention mechanism submodule to obtain a first output feature map; A first extraction unit, configured to extract shallow semantic information of the first output feature map through the shallow semantic extraction submodule to obtain a second output feature map; A second extraction unit, configured to extract deep semantic information of the first output feature map through the deep semantic extraction submodule to obtain a third output feature map; An output unit is used to use the second output feature map and the third output feature map together as a preliminary feature map.

9. The railway rail surface condition detection system based on image semantic segmentation according to claim 8, characterized in that: The feature fusion subsystem includes: A second construction unit is used to construct a feature fusion submodule, wherein the feature fusion submodule includes a fourth convolutional layer and a sixth batch normalization and linear rectification layer; A feature concatenation unit, configured to concatenate the second output feature map and the third output feature map by combining shallow semantic information and deep semantic information to obtain a third concatenated feature map; The adjustment unit is used to adjust the number of channels of the third spliced ​​feature map through the fourth convolution layer and the sixth batch normalization and linear rectification layer in sequence, restore the image resolution, and obtain a multi-scale feature fusion result.

10. The railway rail surface condition detection system based on image semantic segmentation according to claim 9, characterized in that: The detection subsystem comprises: A third construction unit is used to construct a ShuttleNet model, wherein the ShuttleNet model includes a plurality of sequentially connected codecs, wherein the codec includes a coding layer and a decoding layer with a plurality of resolution levels, wherein the plurality of resolution levels of each coding layer are connected by a downsampling operation, and the plurality of decoding layers of each resolution level are connected by an upsampling operation; A fourth construction unit is used to construct a double-layer feature fusion module based on the convolutional attention mechanism submodule, the shallow semantic extraction submodule, the deep semantic extraction submodule and the feature fusion submodule; A definition unit, used to use the multi-scale feature fusion result output by the feature fusion submodule as the output feature of the double-layer feature fusion module; The fifth construction unit is used to embed the double-layer feature fusion module into the output ends of the downsampling operation of each resolution level and the encoding layers of the first four resolution levels in the ShuttleNet model to obtain a railway rail surface state detection model.

Images