Semantic segmentation method and device for train track scene

Abstract

The invention relates to the technical field of semantic segmentation, and provides a semantic segmentation method and device for a train track scene, and the method comprises the steps: obtaining a multi-view image in the train track scene, carrying out the multi-view fusion of the multi-view image, and obtaining a foresight fusion image; inputting the foresight fusion image into a semantic segmentation model to output a semantic segmentation result; the semantic segmentation model comprises an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used for performing down-sampling on the foresight fusion image to obtain down-sampling features; the detail branch is responsible for extracting low-level and shallow-layer detail features in the down-sampling features, the semantic branch is responsible for extracting high-level and deep-layer semantic features in the down-sampling features, and the feature fusion module is responsible for fusing the two features to generate a final semantic segmentation result. The network structure can capture detail information and semantic information of the track scene at the same time, and accuracy and robustness of semantic segmentation of the train track scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a method and device for semantic segmentation of a train track scene. Background Art

[0002] In the fields of automatic train driving, track maintenance, and safety monitoring, accurate understanding and segmentation of train track scenes are crucial. Traditional image processing methods often rely on images from a single perspective. However, due to the complexity and diversity of track scenes, single-perspective images often cannot provide enough information for accurate semantic segmentation. For example, obstacles on the track, signal signs, and the condition of the track itself may not be fully captured and identified due to the limitation of the perspective. In order to solve this problem, the fusion of multi-perspective images has become a research hotspot.

[0003] However, the fusion of multi-view images is not a simple task. Since images from different perspectives may have problems such as lighting differences, perspective deviations, and inconsistent image quality, direct splicing and fusion often result in ghosting, blurring, and information loss in the fused image. Therefore, an effective multi-view image fusion method is needed to overcome these problems and ensure the quality of the fused image. After obtaining a high-quality fused image, the next step is to perform semantic segmentation. However, the accuracy of the semantic segmentation method in the train track scene in the prior art is low, which reduces the accuracy of the semantic segmentation in the train track scene. Summary of the invention

[0004] The present invention provides a method and device for semantic segmentation of a train track scene, which are used to solve the defect that the semantic segmentation method in the train track scene in the prior art has low precision and reduces the accuracy of semantic segmentation in the train track scene.

[0005] The present invention provides a semantic segmentation method for a train track scene, comprising: Acquire multi-view images in a train track scene, and perform multi-view fusion on the multi-view images to obtain a forward-looking fused image; Inputting the forward-looking fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

[0006] According to a semantic segmentation method for a train track scene provided by the present invention, the semantic branch includes a first channel attention module, a first semantic residual unit, a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module and a second upsampling module connected in sequence; The first semantic residual unit, the second semantic residual unit and the third semantic residual unit all include a convolution layer, a normalization layer and an activation layer.

[0007] According to a semantic segmentation method for a train track scene provided by the present invention, the feature fusion module includes a global feature fusion module; The global feature fusion module includes a splicing branch and a global pooling attention branch; The global pooling attention branch includes a first branch, a second branch and a third branch in parallel; the first branch, the second branch and the third branch all include a global average pooling layer, a convolution layer and an upsampling module.

[0008] According to a semantic segmentation method for a train track scene provided by the present invention, the detail branch includes a detail residual unit and a spatial attention module; The detail residual unit includes a depth convolution layer, a normalization layer, an activation layer and a convolution layer.

[0009] According to a semantic segmentation method for a train track scene provided by the present invention, the training step of the semantic segmentation model comprises: Acquire a sample front-view fusion image and a label semantic segmentation result of the sample front-view fusion image, and determine an initial semantic segmentation model; Inputting the sample forward-view fusion image into the initial semantic segmentation model to obtain a predicted semantic segmentation result output by the initial semantic segmentation model; Based on the difference between the label semantic segmentation result and the predicted semantic segmentation result, a target loss is determined, and based on the target loss, parameters of the initial semantic segmentation model are iterated to obtain the semantic segmentation model.

[0010] According to a semantic segmentation method for a train track scene provided by the present invention, the step of obtaining a sample forward-looking fusion image comprises: Acquire multi-view sample images in a train track scene; the multi-view sample images include track images under different lighting conditions and track states; Performing classification and labeling processing on the multi-view sample images to obtain a train track scene labeling image; performing image enhancement processing and geometric transformation on the train track scene marked image in sequence to obtain an enhanced sample image; The enhanced sample image is subjected to multi-view fusion to obtain the sample front-view fusion image.

[0011] The present invention also provides a semantic segmentation device for a train track scene, comprising the following modules: An acquisition unit, used for acquiring multi-view images in a train track scene, and performing multi-view fusion on the multi-view images to obtain a forward-looking fused image; A semantic segmentation unit, used for inputting the front-view fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

[0012] The semantic segmentation method and device of the train track scene provided by the present invention obtains multi-view images in the train track scene, and performs multi-view fusion on the multi-view images to obtain a forward-looking fusion image, and then inputs the forward-looking fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, wherein the semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch, and a feature fusion module. The detail branch is responsible for extracting low-level, shallow detail features, the semantic branch is responsible for extracting high-level, deep semantic features, and the feature fusion module is responsible for fusing these two features to generate the final semantic segmentation result. This network structure can capture the detail information and semantic information of the track scene at the same time, thereby improving the accuracy and robustness of the semantic segmentation of the train track scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is one of the flow charts of the semantic segmentation method for the train track scene provided by the present invention.

[0015] Figure 2 It is a structural schematic diagram of the initial reconstruction unit provided by the present invention.

[0016] Figure 3 It is a structural schematic diagram of the semantic residual unit provided by the present invention.

[0017] Figure 4 It is a structural schematic diagram of the global feature fusion module provided by the present invention.

[0018] Figure 5 It is a schematic diagram of the structure of the detail residual unit provided by the present invention.

[0019] Figure 6 This is the second flow chart of the semantic segmentation method for the train track scene provided by the present invention.

[0020] Figure 7 It is a structural schematic diagram of the semantic segmentation device for the train track scene provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0023] In related technologies, in order to solve the problem that single-view images often cannot provide enough information for accurate semantic segmentation, the fusion of multi-view images has become a research hotspot. By collecting views from different directions and effectively stitching and fusing them, an image containing more complete track scene information can be formed. Such surround-view images not only provide a wider field of view, but also increase the depth and three-dimensionality of the scene, which helps to more accurately identify and segment different objects in the track scene.

[0024] Therefore, an effective multi-view image fusion method is needed to overcome these problems and ensure the quality of the fused image. After obtaining a high-quality fused image, the next step is semantic segmentation. Semantic segmentation is a task that assigns each pixel in an image to a specific category. It requires the algorithm to not only identify the object in the image, but also accurately outline the object's contours and boundaries. In the train track scene, semantic segmentation can help us identify different objects such as tracks, trains, obstacles, signal signs, and provide important information for subsequent tasks such as autonomous driving and track maintenance. In recent years, deep learning has achieved remarkable results in the fields of image processing and computer vision. In particular, convolutional neural networks have greatly improved the accuracy of semantic segmentation of images. Therefore, constructing a semantic segmentation network based on deep learning and inputting the fused track scene image into the network for training is expected to obtain high-precision segmentation results.

[0025] Based on the above problems, the present invention provides a semantic segmentation method for train track scenes. Figure 1 is one of the flow charts of the semantic segmentation method of the train track scene provided by the present invention, such as Figure 1 As shown, the method includes step 110 and step 120.

[0026] Step 110, acquiring multi-view images in a train track scene, and performing multi-view fusion on the multi-view images to obtain a forward-view fusion image; Specifically, multi-perspective images in a train track scene can be obtained. For example, views in the left front, front and right front directions in a train track scene can be collected to obtain multi-perspective images. Here, the multi-perspective image can include track images under different lighting conditions and track states.

[0027] Among them, the multi-perspective images in the train track scene can be collected by using acquisition equipment such as on-board cameras and camera acquisition. The acquisition equipment must be a high-resolution, high-sensitivity RGB camera, with no less than 3 cameras and a stable fixing device. In addition, the acquisition range of each perspective of the left front, front and right front views of the collected train track scene should overlap.

[0028] It should be noted that the multi-view images may include but are not limited to images of categories such as pedestrians, signal signs, catenary poles, road vehicles, trains, and animals.

[0029] After the multi-view images are acquired, they can be fused to obtain a front-view fused image.

[0030] Specifically, the SIFT (Scale-Invariant Feature Transform) algorithm can be used to extract the key points and descriptors of the three views to find matching feature points, and the transformation matrix between images can be estimated using the matching feature points. Each view is transformed into a unified coordinate system for splicing. Then, the overlapping area between adjacent views is determined, and the pixel values ​​of adjacent views are fused using a multi-band fusion method to reduce the stitching seams and brightness differences. Finally, the fused views are combined into a forward-looking fused image containing complete track scene information.

[0031] Step 120, inputting the front-view fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

[0032] Specifically, after obtaining the forward-view fusion image, the forward-view fusion image may be input into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0033] The semantic segmentation model includes an initial reconstruction unit (IRUBlock), a detail branch, a semantic branch and a feature fusion module, wherein the initial reconstruction unit is used to downsample the front-view fusion image to obtain downsampled features.

[0034] Figure 2 is a schematic diagram of the structure of the initial reconstruction unit provided by the present invention, such as Figure 2As shown in the figure, the initial reconstruction unit can be composed of a 3×3 convolutional layer, a BN (Batch Normalization) layer, a ReLU (Rectified Linear Units) activation layer, a spatial channel reconstruction convolutional layer (SCConv) and a 1×1 convolutional layer connected in sequence. After the front-view fusion image obtains a 2-fold down-sampled feature through the initial reconstruction unit, it is subsequently input into the detail branch and the semantic branch to extract features of different levels.

[0035] Among them, the detail branch is used to extract detail information of the downsampled features to obtain the first feature, the semantic branch is used to extract depth information of the downsampled features to obtain the second feature, the feature fusion module is used to fuse the first feature and the second feature to obtain the fused feature, and obtain the semantic segmentation result based on the fused feature.

[0036] It can be understood that the detail branch is responsible for extracting low-level, shallow detail features, the semantic branch is responsible for extracting high-level, deep semantic features, and the feature fusion module is responsible for fusing these two features to generate the final semantic segmentation result. This network structure can capture the detail information and semantic information of the track scene at the same time, improving the accuracy and robustness of the semantic segmentation of the train track scene. The method provided by the embodiment of the present invention effectively solves the limitations of a single perspective in the semantic segmentation of track scenes, and provides strong technical support for automatic train driving and track maintenance.

[0037] Here, the feature fusion of the first feature and the second feature may be performed by concatenating the first feature and the second feature, or by weighting the first feature and the second feature using an attention mechanism and then concatenating them, which is not specifically limited in the embodiment of the present invention.

[0038] It should be noted that the method provided in the embodiment of the present invention can be used for real-time semantic segmentation of traffic scene images for autonomous driving.

[0039] The method provided by the embodiment of the present invention obtains multi-view images in a train track scene, and performs multi-view fusion on the multi-view images to obtain a forward-looking fusion image, and then inputs the forward-looking fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, wherein the semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch, and a feature fusion module. The detail branch is responsible for extracting low-level, shallow detail features, the semantic branch is responsible for extracting high-level, deep semantic features, and the feature fusion module is responsible for fusing these two features to generate the final semantic segmentation result. This network structure can capture both the detail information and semantic information of the track scene, thereby improving the accuracy and robustness of the semantic segmentation of the train track scene.

[0040] Based on the above embodiment, the semantic branch includes a first channel attention module, a first semantic residual unit, a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module and a second upsampling module connected in sequence; The first semantic residual unit, the second semantic residual unit and the third semantic residual unit all include a convolution layer, a normalization layer and an activation layer.

[0041] Specifically, the semantic branch includes a first channel attention module (CAM), a first semantic residual unit (SRU Block), a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module and a second upsampling module, which are connected sequentially.

[0042] Among them, the first semantic residual unit, the second semantic residual unit and the third semantic residual unit all include a convolution layer, a normalization layer and an activation layer.

[0043] Specifically, Figure 3 is a schematic diagram of the structure of the semantic residual unit provided by the present invention, such as Figure 3 As shown in the figure, the first semantic residual unit, the second semantic residual unit and the third semantic residual unit are composed of the first 3×3 convolutional layer (Conv3×3), BN layer, ReLU activation layer, the second 3×3 convolutional layer, BN layer, ReLU activation layer, the first residual addition operation module with 1×1 convolution, the second 3×3 convolutional layer, BN layer, ReLU activation layer, the first 1×1 convolutional layer (Conv1×1), BN layer, the second residual addition operation module, and ReLU activation layer, which are connected in sequence. The semantic branch extracts high-level and deep-level semantic features by using the semantic residual unit to downsample the 2x features output by the initial reconstruction unit.

[0044] The convolution layer here can use a multi-layer convolutional neural network (CNN) with a cascade structure, or a deep neural network (DNN), or a combination of CNN and DNN, etc., which is not specifically limited in the embodiment of the present invention.

[0045] Among them, the upsampling modules are composed of a channel calculation module, a 1×1 convolution layer and a pixel reorganization module connected in sequence. The channel calculation module is provided with an output channel number calculation formula, which is expressed as follows: out c = input c×factor×factor, In the formula, out c is the output channel; input c is the input channel; factor is the upsampling factor, which is set according to the multiple of the output feature map relative to the input feature map. If a feature map twice as large as the input feature map is required, factor=2.

[0046] Based on the above embodiment, the feature fusion module includes a global feature fusion module; The global feature fusion module includes a splicing branch and a global pooling attention branch; The global pooling attention branch includes a first branch, a second branch and a third branch in parallel; the first branch, the second branch and the third branch all include a global average pooling layer, a convolution layer and an upsampling module.

[0047] Specifically, the feature fusion module includes a global feature fusion module (GFFM), wherein the global feature fusion module consists of a splicing branch and a global pooling attention branch.

[0048] Figure 4 is a schematic diagram of the structure of the global feature fusion module provided by the present invention, such as Figure 4 As shown in the figure, the splicing branch is composed of parallel splicing layers and 1×1 convolution layers connected in sequence; the global pooling attention branch includes the first global average pooling layer (Global Pool 1×1) with a pooling kernel of 1×1, the first 1×1 convolution layer and the first upsampling module to form a first sub-branch, the second global average pooling layer (Global Pool 2×2) with a pooling kernel of 2×2, the second 1×1 convolution layer and the second upsampling module to form a second sub-branch, the third global average pooling layer (Global Pool 4×4) with a pooling kernel of 4×4, the third 1×1 convolution layer and the third upsampling module to form a third sub-branch; the output ends of the three sub-branches are passed through the BN layer, the addition operation module and the ReLU activation layer, and then input into the Sigmoid function to calculate the attention weight; the output end of the splicing branch and the attention weight are passed through the multiplication operation module to obtain the final fusion feature.

[0049] It can be understood that by calculating the attention weight and multiplying the output of the splicing branch with the attention weight to obtain the final fusion feature, this module enables the network to pay more attention to the key features in the track scene, suppress irrelevant information, and further improve the accuracy and effect of the semantic segmentation of the train track scene.

[0050] Based on the above embodiment, the detail branch includes a detail residual unit and a spatial attention module; The detail residual unit includes a depth convolution layer, a normalization layer, an activation layer and a convolution layer.

[0051] Specifically, the detail branch includes a detail residual unit (DRU Block) and a spatial attention module (SAM).

[0052] Figure 5 is a schematic diagram of the structure of the detail residual unit provided by the present invention, such as Figure 5 As shown in the figure, the detail residual unit is composed of the first 3×3 deep convolution layer (DConv3×3), BN layer, ReLU activation layer, the first 1×1 convolution layer (Conv1×1), BN layer, the first residual summation module, ReLU activation layer, the second 3×3 deep convolution layer (DConv3×3), BN layer, ReLU activation layer, the second 1×1 convolution layer, BN layer, the second residual summation module, and ReLU activation layer. The detail branch extracts low-level and shallow detail features by downsampling the 2x features output by the initial reconstruction unit through the detail residual unit.

[0053] Based on the above embodiment, the training step of the semantic segmentation model includes: Step 210, obtaining a sample front-view fusion image and a label semantic segmentation result of the sample front-view fusion image, and determining an initial semantic segmentation model; Step 220, inputting the sample front-view fusion image into the initial semantic segmentation model to obtain a predicted semantic segmentation result output by the initial semantic segmentation model; Step 230: determine a target loss based on the difference between the label semantic segmentation result and the predicted semantic segmentation result, and perform parameter iteration on the initial semantic segmentation model based on the target loss to obtain the semantic segmentation model.

[0054] Specifically, in order to better obtain a semantic segmentation model to improve the accuracy and robustness of semantic segmentation of train track scenes, the semantic segmentation model can be trained based on the following steps: First, a sample front-view fusion image and a label semantic segmentation result of the sample front-view fusion image are obtained, and an initial semantic segmentation model is determined.

[0055] After obtaining the sample front-view fusion image, the sample front-view fusion image can be input into the initial semantic segmentation model to obtain the predicted semantic segmentation result output by the initial semantic segmentation model.

[0056] Then, based on the difference between the label semantic segmentation result and the predicted semantic segmentation result, the target loss is determined, and the parameters of the initial semantic segmentation model are iterated based on the target loss, and the initial semantic segmentation model after parameter iteration is used as the semantic segmentation model.

[0057] It can be understood that the greater the difference between the labeled semantic segmentation result and the predicted semantic segmentation result, the greater the target loss; the greater the difference between the labeled semantic segmentation result and the predicted semantic segmentation result, the smaller the target loss.

[0058] In an embodiment of the present invention, the semantic segmentation network training for the train track scene can be developed based on the Linux-Ubuntu 22.04 operating system environment, using pytorch as the deep learning framework, the CUDA version is 12.2, and a single NVIDIA GeForce GTX 4080 Ti GPU is used to complete the model training.

[0059] Set network training parameters: 1) The initial learning rate is set to , the weight decay value is set to ; The dataset size in both training and validation stages is cropped to 512×1024 resolution.

[0060] 2) According to the poly learning rate strategy, set the learning rate in subsequent iterations to: Initial value× ; In the formula, iter is the number of iterations in a subsequent training stage, The number of iterations to find the minimum value of the loss function; At the same time, the momentum=0.9 stochastic gradient descent (SGD) method is set to update the weights and biases of each stage of the network.

[0061] 3) Dynamically set the batch size according to the real-time situation of the training platform. The batch size value is a positive even number between 2 and 8; Set the network training loss function and train the network: During the training process, the output end of the global feature fusion module GFFM is connected to the input end of the segmentation head Seg-Head, and the output result of the output end of the segmentation head Seg-Head is input into the loss function to calculate the target loss function value; wherein, the target loss function can adopt the cross entropy loss (CE Loss) function; in an embodiment of the present invention, the number of iterations of the preheating learning strategy is 1K, and the number of iterations for finding the minimum value of the loss function is at least 100K.

[0062] Based on the above embodiment, the step of obtaining the sample forward-looking fusion image in step 210 includes: Step 211, obtaining multi-view sample images in a train track scene; the multi-view sample images include track images under different lighting conditions and track states; Step 212, classifying and labeling the multi-view sample images to obtain a train track scene label image; Step 213, performing image enhancement processing and geometric transformation on the train track scene marked image in sequence to obtain an enhanced sample image; Step 214, performing multi-view fusion on the enhanced sample image to obtain the sample front-view fused image.

[0063] Specifically, the steps of obtaining the sample forward-looking fusion image are as follows: S1. Collect views from the left front, front and right front directions in the train track scene to obtain multi-view image data, wherein the image data includes track images under different lighting conditions and track states. Specifically, the specific implementation steps of step S1 are as follows: Step S11: Use the on-board camera and the camera to collect train track scene images containing different categories of features, including but not limited to pedestrians, signal signs, catenary poles, road vehicles, trains, animals, etc. The acquisition equipment must be a high-resolution, high-sensitivity RGB camera, with a number of no less than 3, and with a stable fixing device; collect views from the left front, front and right front directions of the train track scene, and the acquisition range of each perspective should overlap; data collection should be carried out under different lighting conditions, weather conditions and track conditions to ensure the comprehensiveness and diversity of the data set.

[0064] Step S12: labeling the collected train track scene images according to the classification standard of the open source railway sensor dataset OSDaR23 to obtain train track scene labeled images.

[0065] Specifically, the labeling processing method is: finely segment different category features in the urban traffic scene image through a standard semantic segmentation tool, and name the segmented category features using a unified naming method to obtain a true segmentation image of the urban traffic scene.

[0066] Step S13: The collected train track scene labeled images are constructed to form an initial semantic segmentation data set.

[0067] S2, expanding the initial image dataset, including introducing open source railway sensor datasets, image enhancement processing (such as brightness and contrast adjustment), geometric transformation (such as rotation and cropping), and simulating images under different weather and lighting conditions, so as to significantly improve the richness and generalization ability of the dataset. Specifically, the specific implementation steps of step S2 are as follows: Step S21, introduce the open source railway sensor dataset OSDaR23, and perform image enhancement processing on the initial image dataset and the introduced open source dataset, including but not limited to brightness adjustment and contrast adjustment, so as to enhance the performance of the image under different lighting conditions and improve the adaptability of the model to lighting changes.

[0068] Step S22: performing geometric transformation on the processed image data set, including but not limited to rotation, cropping and other operations, to increase the diversity of the image so that the model can learn the track features under different perspectives.

[0069] Step S23, simulating images under different weather and lighting conditions, including sunny, cloudy, rainy, foggy days and lighting conditions at different time periods, to further enrich the data set and improve the segmentation performance of the model under different weather and lighting environments.

[0070] Step S24: The expanded labeled images of all train track scenes are constructed to form a final semantic segmentation dataset.

[0071] S3, the three directional views included in the expanded data set are multi-view fused to form a forward-looking fusion image containing complete track scene information. The forward-looking fusion image is split into a test set, a training set, and a validation set according to a certain ratio. Specifically, the specific implementation steps of step S3 are as follows: Step S31: Use SIFT feature detection algorithm to extract key points and descriptors of three views to find matching feature points, use matching feature points to estimate the transformation matrix between images, and transform each view into a unified coordinate system for splicing.

[0072] Step S32: determine the overlapping area between adjacent views, and fuse the pixel values ​​of adjacent views using a multi-band fusion method to reduce the stitching seams and brightness differences.

[0073] Step S33: merge the fused views into a forward-looking fused image containing complete track scene information.

[0074] Step S34: split the front-view fusion image into a test set, a training set, and a validation set according to a ratio of 2:7:1.

[0075] The method provided in the embodiment of the present invention collects views from the left front, front and right front directions in the train track scene, and performs multi-view fusion to form a forward fusion image containing complete track scene information. In addition, the richness and generalization ability of the dataset are significantly improved by introducing open source railway sensor datasets, image enhancement processing, geometric transformation, and simulating images under different weather and lighting conditions.

[0076] Based on any of the above embodiments, Figure 6 FIG. 2 is a flow chart of the semantic segmentation method of the train track scene provided by the present invention. Figure 6 As shown, the method includes: The first step is to obtain multi-view images in the train track scene and perform multi-view fusion on the multi-view images to obtain a forward-looking fusion image.

[0077] In the second step, the front-view fusion image is input into the semantic segmentation model to obtain the semantic segmentation result output by the semantic segmentation model.

[0078] Among them, the semantic segmentation model includes an initial reconstruction unit (IRU Block), a detail branch (Detail Branch), a semantic branch (Semantic Branch) and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a semantic segmentation result.

[0079] Here, the semantic branch includes a first channel attention module (CAM), a first semantic residual unit (SRU Block), a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module (Up Block), and a second upsampling module, which are sequentially connected. The first semantic residual unit, the second semantic residual unit, and the third semantic residual unit all include a convolution layer, a normalization layer, and an activation layer.

[0080] Among them, the stride in the first semantic residual unit is 1, the stride in the second semantic residual unit is 2, and the stride in the third semantic residual unit is 2.

[0081] Here, the feature fusion module includes a global feature fusion module (GFFM), an upsampling module, and a segmentation head Seg-Head. The global feature fusion module includes a splicing branch and a global pooling attention branch; The global pooling attention branch includes a parallel first branch, a second branch, and a third branch; the first branch, the second branch, and the third branch all include a global average pooling layer, a convolutional layer, and an up-sampling module (Up Block).

[0082] Here, the detail branch includes a detail residual unit and a spatial attention module (SAM), and the detail residual unit includes a deep convolutional layer, a normalization layer, an activation layer, and a convolutional layer.

[0083] The semantic segmentation device for the train track scene provided by the present invention is described below. The semantic segmentation device for the train track scene described below and the semantic segmentation method for the train track scene described above can be referenced to each other.

[0084] Based on any of the above embodiments, the present invention provides a semantic segmentation device for a train track scene. Figure 7 is a schematic diagram of the structure of the semantic segmentation device for the train track scene provided by the present invention, such as Figure 7 As shown, the device comprises: An acquisition unit 710 is used to acquire multi-view images in a train track scene, and perform multi-view fusion on the multi-view images to obtain a forward-view fusion image; A semantic segmentation unit 720, configured to input the front-view fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

[0085] The device provided by the embodiment of the present invention obtains multi-view images in a train track scene, and performs multi-view fusion on the multi-view images to obtain a forward-looking fusion image, and then inputs the forward-looking fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, wherein the semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch, and a feature fusion module. The detail branch is responsible for extracting low-level, shallow detail features, the semantic branch is responsible for extracting high-level, deep semantic features, and the feature fusion module is responsible for fusing these two features to generate the final semantic segmentation result. This network structure can capture both the detail information and semantic information of the track scene, thereby improving the accuracy and robustness of the semantic segmentation of the train track scene.

[0086] Based on any of the above embodiments, the semantic branch includes a first channel attention module, a first semantic residual unit, a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module, and a second upsampling module connected in sequence; The first semantic residual unit, the second semantic residual unit and the third semantic residual unit all include a convolution layer, a normalization layer and an activation layer.

[0087] Based on any of the above embodiments, the feature fusion module includes a global feature fusion module; The global feature fusion module includes a splicing branch and a global pooling attention branch; The global pooling attention branch includes a first branch, a second branch and a third branch in parallel; the first branch, the second branch and the third branch all include a global average pooling layer, a convolution layer and an upsampling module.

[0088] Based on any of the above embodiments, the detail branch includes a detail residual unit and a spatial attention module; The detail residual unit includes a depth convolution layer, a normalization layer, an activation layer and a convolution layer.

[0089] Based on any of the above embodiments, it further includes a training unit, wherein the training unit is specifically used to: A sample image acquisition unit is used to acquire a sample front-view fusion image and a label semantic segmentation result of the sample front-view fusion image, and determine an initial semantic segmentation model; A prediction unit, used for inputting the sample forward-view fusion image into the initial semantic segmentation model to obtain a predicted semantic segmentation result output by the initial semantic segmentation model; A parameter iteration unit is used to determine a target loss based on a difference between the label semantic segmentation result and the predicted semantic segmentation result, and to perform parameter iteration on the initial semantic segmentation model based on the target loss to obtain the semantic segmentation model.

[0090] Based on any of the above embodiments, the sample image acquisition unit is specifically used to: Acquire multi-view sample images in a train track scene; the multi-view sample images include track images under different lighting conditions and track states; Performing classification and labeling processing on the multi-view sample images to obtain a train track scene labeling image; performing image enhancement processing and geometric transformation on the train track scene marked image in sequence to obtain an enhanced sample image; The enhanced sample image is subjected to multi-view fusion to obtain the sample front-view fusion image.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for a train track scene, characterized in that: include: Acquire multi-view images in a train track scene, and perform multi-view fusion on the multi-view images to obtain a forward-looking fused image; Inputting the forward-looking fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

2. The method for semantic segmentation of a train track scene according to claim 1, characterized in that: The semantic branch includes a first channel attention module, a first semantic residual unit, a second channel attention module, a second semantic residual unit, a third channel attention module, a third semantic residual unit, a first upsampling module and a second upsampling module connected in sequence; The first semantic residual unit, the second semantic residual unit and the third semantic residual unit all include a convolution layer, a normalization layer and an activation layer.

3. The method for semantic segmentation of a train track scene according to claim 1, characterized in that: The feature fusion module includes a global feature fusion module; The global feature fusion module includes a splicing branch and a global pooling attention branch; The global pooling attention branch includes a first branch, a second branch and a third branch in parallel; the first branch, the second branch and the third branch all include a global average pooling layer, a convolution layer and an upsampling module.

4. The method for semantic segmentation of a train track scene according to claim 1, characterized in that: The detail branch includes a detail residual unit and a spatial attention module; The detail residual unit includes a depth convolution layer, a normalization layer, an activation layer and a convolution layer.

5. The method for semantic segmentation of a train track scene according to any one of claims 1 to 4, characterized in that: The training steps of the semantic segmentation model include: Acquire a sample front-view fusion image and a label semantic segmentation result of the sample front-view fusion image, and determine an initial semantic segmentation model; Inputting the sample forward-view fusion image into the initial semantic segmentation model to obtain a predicted semantic segmentation result output by the initial semantic segmentation model; Based on the difference between the label semantic segmentation result and the predicted semantic segmentation result, a target loss is determined, and based on the target loss, parameters of the initial semantic segmentation model are iterated to obtain the semantic segmentation model.

6. The method for semantic segmentation of a train track scene according to claim 5, characterized in that: The obtaining of the sample forward-looking fusion image comprises: Acquire multi-view sample images in a train track scene; the multi-view sample images include track images under different lighting conditions and track states; Performing classification and labeling processing on the multi-view sample images to obtain a train track scene labeling image; performing image enhancement processing and geometric transformation on the train track scene marked image in sequence to obtain an enhanced sample image; The enhanced sample image is fused from multiple perspectives to obtain the sample front-view fused image.

7. A semantic segmentation device for a train track scene, characterized in that: include: An acquisition unit, used for acquiring multi-view images in a train track scene, and performing multi-view fusion on the multi-view images to obtain a forward-looking fused image; A semantic segmentation unit, used for inputting the front-view fusion image into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model; The semantic segmentation model includes an initial reconstruction unit, a detail branch, a semantic branch and a feature fusion module; the initial reconstruction unit is used to downsample the front-view fusion image to obtain a downsampled feature; the detail branch is used to extract detail information from the downsampled feature to obtain a first feature; the semantic branch is used to extract depth information from the downsampled feature to obtain a second feature; the feature fusion module is used to fuse the first feature and the second feature to obtain a fused feature, and obtain the semantic segmentation result based on the fused feature.

Images