A Detection Method for Motor Train Oil Traces Based on Progressive Context Understanding Network

Abstract

The present invention relates to a method for detecting oil stains on a bullet train based on a progressive context understanding network, which includes the following steps: obtaining a two-dimensional image of the internal structure of the bullet train using a high-definition camera, screening the data set, and dividing it into normal image data and data with oil stain images; in the training stage, performing pixel-level annotation on the images, marking the oil stain areas in the images and setting their pixel values to 255, and setting the remaining non-oil stain areas to 0, which is the mask of the image; respectively preprocessing the original image and its mask image, including data augmentation such as random flipping and normalization operations; inputting the preprocessed data into the progressive context understanding network for feature learning to obtain a binary image that may include the oil stain area. Finally, using a fully connected conditional random field as a post-processing operation to further optimize the output binary image to obtain a more accurate result. The present invention can accurately detect the position of the oil stain in the image of the internal structure of the bullet train.

Description

Technical Field

[0001] The present invention belongs to the field of abnormal detection of bullet trains, and particularly relates to a method for detecting oil traces of bullet trains based on a progressive context understanding network. Background Art

[0002] The railway industry has achieved greater and greater achievements in recent years, especially high-speed trains and bullet trains. There are a large number of high-speed trains and bullet trains running in the country every day. Due to their high speed and convenience, they are also welcomed by residents and have become one of their main means of travel. However, since bullet trains are in a high-speed operation mode for a long time and in a complex working environment, they may have certain failures or abnormalities during operation, resulting in various losses. Generally speaking, common abnormalities in bullet trains include bolt loss, cable detachment, internal foreign object entanglement, and oil leakage, etc. Therefore, in order to ensure the normal operation of the train and avoid accidental losses, it is necessary to detect bullet trains to discover possible abnormalities.

[0003] Among so many abnormal faults of bullet trains, the detection of oil leakage is difficult to detect due to its inherent properties. The shape of the oil trace formed by oil leakage is irregular and cannot be expressed by existing shapes. In addition, the sizes of different oil trace areas vary greatly. Small ones may have only dozens of pixels on the image, while large ones may occupy most of the image area. In the early stage of technological development, the method of manual detection was used to judge whether there was an abnormality. This method was time-consuming and laborious, with extremely low efficiency, and human resources were easily affected by subjective and objective factors, resulting in inaccurate detection results. Fortunately, due to the rapid progress and high-speed development of computer vision and deep learning technologies, it is very likely to automatically detect bullet train abnormalities through a computer. In this field of abnormal detection, common methods include methods based on detection, localization, and then classification, and there are also methods based on image segmentation to output the abnormal position.

[0004] At present, due to the properties of the oil trace itself, there are few methods for detecting oil traces of bullet trains. Therefore, there is an urgent need for a method with good performance and suitability to detect oil traces of bullet trains, so as to avoid the losses and damages that may be caused by bullet train oil leakage. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to provide a suitable method with better effect for detecting the position of oil traces in the internal structure image of a bullet train.

[0006] In order to solve the above technical problem, the technical solution proposed by the present invention is: a method for detecting oil traces of bullet trains based on a progressive context understanding network: the method includes the following steps:

[0007] Step 1: Take two-dimensional image data of the interior structure of the EMU through a high-definition camera, screen the data to obtain normal images and abnormal images with oil stains. Subsequently, perform pixel-level annotation on the images to obtain the mask of the images.

[0008] Step 2: Preprocess the input original image and mask image respectively, including image enhancement and normalization operations.

[0009] Step 3: Input the preprocessed original image and its mask image into the progressive context understanding network to learn features, extract features, and then calculate the loss with the mask image through the loss function designed by us to adjust the weight parameters in the network, optimize the network, and obtain the optimal network model. Finally, the output of the model is a binary image B that may have an oil stain area.

[0010] Step 4: Adopt the fully connected conditional random field (CRF) algorithm as a post-processing operation to further optimize the binary image B output by the network, so as to obtain a more accurate oil stain detection result.

[0011] Furthermore, in the first step, during the training stage, first perform pixel-level annotation on the original image based on LabelMe, set the pixel values of the pixel points in the oil stain area of the annotated image to 255, and the pixel values of the pixel points in the remaining non-oil stain areas to 0, that is, obtain the mask of the image; during testing, this step is not required.

[0012] Furthermore, in the second step, during training, perform data augmentation operations on the original image and mask image respectively, including random flipping, adjust the resolution of the images to the same fixed value, and then perform normalization operations respectively. Subtract the mean from each of the three channels of the original image and divide by the standard deviation, and divide the mask image by 255; during testing, only resize the image to the same size as during training and perform the normalization operation.

[0013] Furthermore, in the third step, input the original image and mask together into the progressive context understanding network, learn features from the original image, extract features, output a binary map that may have an oil stain area, calculate the loss with the mask based on the designed loss function, and backpropagate to adjust the model parameters, and continuously train until an optimal model is obtained. This network consists of multiple sub-network modules, including an upsampling enhancement module (UAM), a feature refinement model (RFM), a pyramid context fusion module (PCFM), and a deconvolution upsampling module (DM). In addition, the loss function designed by us consists of cross-entropy loss and weighted mean absolute error loss.

[0014] Furthermore, in the fourth step, optimize the oil stain area in the image through the fully connected conditional random field, and more accurately locate the boundary area of the oil stain.

[0015] The present invention provides a method for detecting oil stains on high-speed trains based on a progressive context understanding network, which solves the problem in the prior art that it is difficult to detect oil leakage in high-speed trains by detecting whether there are oil stains on the internal components of high-speed trains. Moreover, it can adapt to the complex environment inside the train and has a good detection effect on various oil stains. Therefore, both the accuracy and recall rate of oil stain detection have good performance.

[0016] The innovation points of the present invention mainly include the following aspects:

[0017] 1) The present invention first proposes to solve the oil leakage problem of trains based on the oil stains on the internal components of trains, and specifically proposes a progressive context understanding network as a solution to this problem. Among them, upsampling enhancement module (UAM), feature refinement module (RFM), pyramid context fusion module (PCFM) and deconvolution upsampling module (DM) are proposed in this network;

[0018] 2) The present invention designs a loss function, namely cross-entropy loss and weighted mean absolute error loss, to train and optimize the above-mentioned progressive context understanding network, which can strengthen the model's learning of the features of the oil stain area when the difference between the oil stain area and the background area is large, so as to better detect the oil stains;

[0019] 3) The detection method of the present invention can detect oil stains in the complex internal environment of high-speed trains, and can also detect various oil stain areas, such as oil stains of different sizes, and has good accuracy and recall rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below with reference to the accompanying drawings.

[0021] Figure 1 is an image of the interior of a high-speed train containing oil stains in different situations. Among them, (a) is a schematic diagram of an irregular oil stain image, (b) is a schematic diagram of the area sizes of different oil stains, and (c) is a schematic diagram of a very uneven distribution of oil stains and the background.

[0022] Figure 2 is the oil stain detection result. Among them, (a) and (b) include schematic diagrams of detecting small-area and large-area oil stains and detecting multiple oil stain areas in one picture at the same time.

[0023] Figure 3 is the flowchart of the context understanding network.

[0024] Figure 4It is a flowchart of a sub-module in the context understanding network, where (a) is the upsampling enhancement module (UAM), (b) is the feature refinement module (FRM), (c) is the pyramid context fusion module (PCFM), and (d) is the deconvolution module (DM).

[0025] Figure 5 It is a flowchart of the present invention. Detailed implementation manners

[0026] The present invention will be further illustrated below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent modifications made by those skilled in the art fall within the scope defined by the appended claims of this application.

[0027] Embodiment

[0028] The images to be detected are pictures of devices at different orientations inside a bullet train taken by a camera mounted on a track. Figure 1 The following are several typical cases of oil stains in this example. As can be seen from Figure (a), the shapes of the oil stains are various and without any pattern, and cannot be described by any shape. As can be seen from Figures (b) and (c), the area sizes of the oil stains themselves are also very different on different images. In addition, their size distributions and backgrounds are also very different. Some small ones are only dozens of pixel points in size, while large oil stains occupy most of the image area.

[0029] This embodiment trains 570 images and tests 500 images in a manner similar to Figure 1 the image distribution shown, and these images are collected by a camera. As Figure 5 shown, its oil stain detection method includes the following steps:

[0030] Step 1: In the training stage, perform pixel-level annotation on the collected images, and then set the pixel values of the pixel points in the oil stain area of the image to 255, and set the pixel values of the remaining non-oil stain areas to 0, so as to obtain the mask of the image. In the testing stage, there is no need to annotate the images.

[0031] Step 2: In the training stage, some preprocessing needs to be done on the original image and the mask. For example, some data augmentation operations need to be performed on the original image, such as random flipping, resizing to a fixed value, and then normalizing the original image and the mask; during testing, only resize and normalize the original image.

[0032] Step 3: In the training stage, input the original image and the mask into the progressive context understanding network simultaneously, as Figure 3 shown. The specific steps are as follows:

[0033] (1)Extract features at different stages based on the backbone network, and then obtain features of different size resolutions. Channel splitting, convolution, and splicing are performed on these features for dimensionality reduction.

[0034] (2)Feature enhancement is performed on images of different resolutions, and features of adjacent sizes are further refined. For example, for features of size 18*18 and features of size 36*36, first, based on the upsampling enhancement module (UAM) as shown in Figure 4 (a), the smaller image is upsampled to the same size as the larger feature map, and then the two are spliced together, and convolution is performed on them from different aspects to enhance the features. Then, based on the feature refinement module (FRM) as shown in Figure 4 (b), the upsampled and enhanced features and the original-size features are used to further refine the features while reducing the dimension.

[0035] (3)As shown in Figure 3 , the feature map is learned and enhanced based on the upsampling enhancement module and the feature enhancement module in sequence, and the context feature maps are gradually fused to obtain more representative features.

[0036] (4)After enhancing the context of features of different scales, based on the pyramid context fusion module (PCFM) as shown in Figure 4 (c), the enhanced features are fused pairwise to obtain four features with stronger representational ability at different scales. Then, based on the deconvolution module (DW), the features of the largest scale are upsampled and learned to obtain two features of a larger scale. Secondly, a result is predicted for each of these features.

[0037] (5)Finally, the results of the above predictions are combined to fuse and obtain a more representative result.

[0038] It should be noted that we have the following loss function designed by us:

[0039] L = L b + L m (1)

[0040] Among them, L b is the binary cross-entropy loss, and L m is the weighted average absolute value loss, and its specific expression is as follows:

[0041]

[0042] Among them, M is the number of training sets, N is the number of all pixel points, O and G respectively represent the predicted binary oil trace image and the labeled mask image, O p and G p represent the oil traces in the image, On and G n represents the background in the image, and α is a hyperparameter.

[0043] In the training stage, the results predicted for these different-scale features are supervised and learned based on the above loss function. During testing, we load the trained model, and the final predicted result is the fused result.

[0044] Step 4: Optimize the binary image predicted by the above model through a fully connected conditional random field (CRF) to more accurately locate the boundary region of the oil stain. Figure 2 The oil stain detection results of some images are shown.

[0045] The present invention is not limited to the specific technical solutions described in the above embodiments. In addition to the above embodiments, the present invention may have other implementation manners. For those skilled in the art, any technical solutions formed by making any modifications, equivalent replacements, improvements, etc. within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting oil stains on bullet trains based on a progressive context understanding network, characterized in that, the method comprises the following steps: Step 1: Obtain two-dimensional image data by photographing the internal structure of a bullet train with a high-definition camera, screen the data to obtain normal images and abnormal images with oil stains, and then perform pixel-level annotation on the images to obtain the mask of the images; Step 2: Preprocess the input original image and mask image respectively, including image enhancement and normalization operations; Step 3: Input the preprocessed original image and its mask image into the progressive context understanding network to learn features, extract features, and then calculate the loss with the mask image through a loss function to adjust the weight parameters in the network, optimize the network, and obtain the optimal network model. Finally, the output of the model is a binary image B of the area that may have oil stains; Step 4: Use the fully connected conditional random field algorithm as a post-processing operation to further optimize the binary image B output by the network to obtain a more accurate oil stain detection result; In the said Step 3, the progressive context understanding network is composed of multiple sub-network modules, including an upsampling enhancement module, a feature refinement model, a pyramid context fusion module, and a deconvolution upsampling module. In addition, the loss function is composed of cross-entropy loss and weighted mean absolute error loss; The specific steps are as follows: (1) Extract features at different stages based on the backbone network, and then obtain features with different size resolutions. Perform channel segmentation, convolution, and splicing on these features to reduce the dimension; (2) Strengthen the features of images with different resolutions, and further refine the features of adjacent sizes; (3) Learn and strengthen the feature maps based on the upsampling enhancement module and the feature refinement model in turn, and progressively fuse the context feature maps to obtain more representative features; (4) After enhancing the context of features at different scales, based on the pyramid context fusion module, fuse the enhanced features pairwise to obtain four features with stronger representation capabilities at different scales. Then, based on the deconvolution module, perform upsampling learning on the features of the largest scale to obtain two features of a larger scale; Secondly, predict a result for each of these features; (5) Finally, integrate the above predicted results to fuse and obtain a more representative result.

2. A method for detecting oil stains on bullet trains based on a progressive context understanding network according to claim 1, characterized in that: In the said Step 1, in the training stage, perform pixel-level annotation on the images through the LabelMe software. Then, set the pixel values of the pixel points in the oil stain area marked in the image to 255, and set the pixel values of the pixel points in other non-oil stain areas to 0, that is, obtain the mask of the image. In the test, this step is not required.

3. A method for detecting oil stains on bullet trains based on a progressive context understanding network according to claim 1, characterized in that: In the second step, during the training phase, data augmentation operations are performed on the original image and the mask image respectively, including randomly flipping, adjusting the image resolution to a unified size, and then performing normalization operations respectively. For the three channels of the original image, the mean is subtracted from each channel and then divided by the standard deviation, and for the mask image, it is divided by 255. During testing, only resizing and normalization of the image are required.

4. A method for detecting oil traces on a bullet train based on a progressive context understanding network according to claim 1, characterized in that: In the fourth step, the oil trace area in the image is optimized by a fully connected conditional random field to more accurately locate the boundary area of the oil trace.

Images