A method for detecting oil stains on electric trains based on a multi-level feature aggregation network

Abstract

The present invention discloses a method for detecting oil stains on a motor vehicle based on a multi-level feature aggregation network. The method comprises the following steps: obtaining an original image according to the internal structure of the motor vehicle, dividing the original image into a normal image and an image with oil stains, and manually annotating the image; obtaining a corresponding mask image for each annotated image; preprocessing the original image and the corresponding mask image, inputting the image into a multi-level feature aggregation network model to extract features, enhance features, and fuse multiple features; outputting an image of oil stain distribution; and continuously adjusting the network parameters based on a loss function and a stochastic gradient descent algorithm to obtain the most suitable multi-level feature aggregation network model; the model comprises at least a cross-layer feature attention module, a multi-level refined feature module, and a cross-layer feature enhancement module; and performing post-processing based on a fully connected conditional random field to optimize the boundary area in the image and outputting a binary map of oil stain distribution, thereby providing an effective and suitable method for locating the position of oil stains on a train.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent computers and data detection, and in particular relates to a motor vehicle oil trace detection method based on a multi-level feature aggregation network. Background Art

[0002] High-speed train technology has developed rapidly in recent years, driven by rapid advances in science and technology. It continues to gain popularity due to its convenience and insensitivity to environmental factors such as weather. However, under long-term operation and complex, ever-changing operating environments, there is a potential for sudden failures, such as missing bolts, broken loose wires, cable drops and disconnections, and oil leaks. Any minor fault occurring at high speeds can pose a significant threat to the safety of the train and its passengers. Therefore, ensuring the normal and safe operation of trains and promptly detecting any faults are essential and extremely important.

[0003] Among the many common train faults, while oil stain detection can be used to identify oil leaks, its irregular shape and fluidity make it difficult to describe in words. Furthermore, the size of oil stains varies greatly. Small oil stains may not be readily apparent in images, making them easily overlooked and leading to missed detections. Large oil stains may be widespread but not entirely continuous, resulting in incomplete detection. Furthermore, some oil stains may resemble normal areas in the image, leading to false detections. Early manual detection methods were laborious, inefficient, and ineffective. These methods were susceptible to numerous factors, making it difficult to achieve satisfactory results. However, the emergence and significant advancements of computer vision and deep learning have made automated detection of train faults possible.

[0004] Currently, there are few methods for detecting oil leaks on EMUs, considering the nature of oil stains and the specificity of the problem. Therefore, there is an urgent need for an effective and appropriate method to detect oil stains and identify oil leaks, thereby preventing train safety issues caused by oil leaks. Summary of the Invention

[0005] The present invention is aimed at the problems existing in the prior art and provides a method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network. First, the original image is obtained according to the internal structure of the motor vehicle, and it is divided into a normal image and an image with oil stains, and manually annotated. Each image annotation obtains a corresponding mask image, and the value of the oil stain part in the mask is set to 255, and the value of the normal part in the mask is set to 0; after preprocessing the original image and its corresponding mask image, the original image and the corresponding mask are input into a multi-level feature aggregation network model to extract features, enhance features, and fuse various features, capture representative features, and output relevant features. The method is based on a binary map of oil stain distribution, and the gap between the binary map and the mask image data annotated with the original image is calculated based on the loss function. The network parameters are continuously adjusted based on the stochastic gradient descent algorithm, and the network weights are continuously updated. That is, samples are randomly selected each time to calculate the loss function, and the gradient is obtained to update the network parameters to obtain the most suitable multi-level feature aggregation network model; the multi-level feature aggregation network model at least includes a cross-layer feature attention module, a multi-level refinement feature module, and a cross-layer feature enhancement module; finally, post-processing is performed based on a fully connected conditional random field to optimize the boundary area in the image and output a binary map of oil stain distribution, providing an effective and suitable method for locating the position of oil stains in trains.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a motor vehicle oil stain detection method based on a multi-level feature aggregation network, the method comprising the following steps:

[0007] S1, image acquisition and annotation: obtain the original image based on the internal structure of the train, organize the original image, divide it into normal images and images with oil stains, and use LabelMe software for manual annotation, that is, each image will generate a corresponding mask image;

[0008] S2, data preprocessing: preprocessing the input original image and the corresponding mask image obtained in step S1 respectively, wherein the preprocessing includes at least image enhancement and normalization operations;

[0009] S3, multi-level feature aggregation: the original image pre-processed in step S2 and its corresponding mask image are input into the multi-level feature aggregation network to extract features, multi-level enhanced features and fused features, and predict an image of the oil stain distribution; based on the loss function, the gap between the distribution map and the mask image data annotated with the original image is calculated, and the network parameters are continuously adjusted based on the stochastic gradient descent algorithm, and the network weights are continuously updated, that is, each time a sample is randomly selected to calculate the loss function, the gradient is obtained to update the network parameters, and the most appropriate multi-level feature aggregation network model is obtained; the multi-level feature aggregation network model at least includes a cross-layer feature attention module, a multi-level refinement feature module, and a cross-layer feature enhancement module;

[0010] S4, fully connected conditional random field optimization: using a fully connected conditional random field algorithm combined with the original image to optimize the predicted image of the oil stain distribution; the optimization specifically involves further optimizing the boundaries of the oil stain image output by the network and improving the misclassified areas based on the relationship between pixels in the original image;

[0011] S5, oil trace distribution binary map output: After the optimization of step S4, the oil trace distribution binary map is obtained.

[0012] Compared with the prior art, the technical advantages of the present invention are:

[0013] (1) The present invention realizes computer automatic detection of whether there are oil stains on key parts of the EMU based on the algorithm to further discover whether there is an oil leakage problem in the EMU. The automatic implementation of the algorithm not only saves a lot of manpower and material resources, but also has a good detection effect for the complex and changeable environment inside and outside the train.

[0014] (2) The present invention proposes a multi-level feature aggregation network as a solution to this problem. The network model proposes a cross-layer feature attention module (CLFA), a multi-level refined feature module (MRF), and a cross-layer feature enhancement module (CLFE). The enhanced features are expressed more clearly, the features are more representative, and the detection effect is better.

[0015] (3) In the multi-level feature aggregation network, the method of the present invention proposes a loss function that combines weighted binary cross entropy loss and improved mean absolute error loss to perform algorithm iteration and parameter adjustment and optimization on the multi-level feature aggregation network, which can promote the network's learning of the oil stain area under the condition of extremely unbalanced distribution between the foreground area, i.e., the oil stain, and the background area.

[0016] (4) The detection method of the present invention can better detect oil stains under complex and changeable internal and external conditions of the train, and also has good detection effects on oil stains of different shapes and sizes, and has good algorithm performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the distribution of various oil stains in the EMU;

[0018] Figure 2 This is a flowchart of the steps of a motor vehicle oil stain detection method based on a multi-level feature aggregation network of the present invention;

[0019] Figure 3 This is a structural diagram of the multi-level feature aggregation network in step S3 of the method of the present invention;

[0020] Figure 4a-4cIt is the structural diagram of the submodule in the multi-level feature aggregation network, where

[0021] Figure 4a Schematic diagram of the cross-layer feature attention module (CLFA);

[0022] Figure 4b Schematic diagram of the multi-level refined feature module (MRF) structure;

[0023] Figure 4c Schematic diagram of the structure of the cross-layer feature enhancement module (CLFE);

[0024] Figure 5 This is a partial result diagram of oil stain detection in Example 2 of the present invention, where

[0025] Figure 5 (a) is a schematic diagram of oil stain areas of various sizes;

[0026] Figure 5 (b) is a schematic diagram of detecting multiple oil stain areas in one image. DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0028] Example 1

[0029] A method for detecting oil stains on EMUs based on a multi-level feature aggregation network is designed for component diagrams at different locations inside the EMU, such as Figure 1 As shown, Figure 1 That is, the oil stains on components in different positions inside the EMU. It can be seen that the shape of the oil stains is very irregular and cannot be described in words. Moreover, the sizes of different oil stains are also very different. Some small ones are easily overlooked, while the large oil stain areas have poor continuity. In addition, the appearance or color of some oil stains are very similar to the background.

[0030] The oil stain detection method of motor vehicle based on multi-level feature aggregation network, such as Figure 2 As shown, the method includes the following steps:

[0031] Step S1: Image acquisition and annotation:

[0032] The internal structure of the train is captured by a camera, and the images are sorted and divided into normal images and images with oil stains. The data is manually annotated using LabelMe software. That is, each training image will generate a corresponding mask image. The pixel values ​​at the positions corresponding to the oil stain areas in the original image on the mask image are set to 255, and the rest of the normal areas are set to 0. During the testing phase, there is no need to annotate the images.

[0033] Step S2, data preprocessing:

[0034] Preprocessing the input original image and the corresponding mask image obtained in step S1, respectively, wherein the preprocessing includes at least image enhancement and normalization operations;

[0035] The original data and its corresponding mask data are randomly flipped horizontally for data augmentation. The input data scale is set to a fixed size. The original data and its mask data are then normalized. That is, the three channels of the original data are subtracted from the mean and standard deviation based on the statistics of the ImageNet dataset, and divided by the standard deviation. The mask data is divided by 255 to normalize to 0-1. During testing, the original image only needs to be resized and normalized to keep consistent with the training time.

[0036] Step S3, multi-level feature aggregation:

[0037] The original image preprocessed in step S2 and its corresponding mask image are input into the multi-level feature aggregation network, such as Figure 3 As shown in the figure, the input of the network is a three-channel image. First, a skeleton network such as ResNext101-32x4d is used to extract multi-level image features. The DR module reduces the number of channels of each feature. Then, based on the cross-layer feature attention module, the multi-level refinement module, and the cross-layer feature enhancement module, the features of each level are enhanced, including low-level local features and high-level semantic features. Finally, an image of the oil stain distribution is predicted.

[0038] Based on the backbone network, features of different levels and scales are extracted from the image. After the DR module reduces the dimension, the channels of these features are evenly divided. Then, feature extraction and dimensionality reduction are performed based on convolution, batch normalization and activation function respectively. Finally, the features are spliced ​​together.

[0039] The multi-level feature aggregation network model at least includes a cross-layer feature attention module, a multi-level refinement feature module, and a cross-layer feature enhancement module. Figures 4a-4c As shown, where:

[0040] The reason why the two features at higher levels are strengthened based on features at multiple levels is that different layers have diverse object information due to different ranges of receptive fields. Figure 4aAs shown, the CLFA_1 module primarily generates an attention mechanism graph based on the interaction of features at four levels. This attention mechanism graph is then used to enrich features at the third level. Similarly, the CLFA_2 module works the same way, but this module enhances information from features at the highest level. Essentially, the cross-layer feature attention module transforms features from multiple levels to the same scale using convolution, then fuses them by adding them together. The corresponding weights are calculated and multiplied by the corresponding features, and finally added to the original features. This method enhances the expressiveness of features and makes them clearer.

[0041] Combining low-level features with high-level features can enhance image features at different scales, making the features more effective and obtaining more representative features. Figure 4b As shown in the figure, the input of the multi-level refined feature module is the upsampling result of the larger-scale features of the lower level and the smaller-scale features of the higher level, or the splicing of multiple upsampling results. First, the two inputs are spliced ​​together, and then features are extracted from different convolution methods such as separable convolution and grouped convolution. The results are then added to the lower-level features, and then enhanced based on different convolution methods such as horizontal convolution, vertical convolution, etc. Finally, they are spliced ​​together and dimensionality reduction is performed based on convolution. The low-level local features and high-level global features are combined in a variety of ways to make the features more representative.

[0042] Enhance low-level features based on other high-level features, such as Figure 4c As shown in Figure 1, the cross-layer feature enhancement module upsamples the features of the other three higher layers to the same scale as the lowest layer features, concatenates them and feeds them into the MRF module for enhancement. Based on the high-level semantic features, the low-level detail features are enhanced to make them richer and have some semantic information.

[0043] The oil stain distribution is predicted based on high-level and low-level features, and then the two results are combined to obtain a more representative distribution map. In this process, the gap between the binary image and the mask image data annotated with the original image is calculated based on the loss function. The network parameters are continuously adjusted based on the stochastic gradient descent algorithm, and the network weights are continuously updated. That is, each time a sample is randomly selected to calculate the loss function, the gradient is obtained to update the network parameters, and the most appropriate multi-level feature aggregation network model is obtained. The loss function designed in this embodiment is as follows:

[0044] L=α·L bce +β·L m

[0045] Among them, α and β are the hyperparameters of adjustment, L bce is the binary cross entropy loss, L m is the improved mean absolute value loss, which is expressed as follows:

[0046]

[0047] Where M is the number of images in the training set, σ is the sigmoid function, and p i is the i-th predicted image, g i is the i-th labeled image, sum(g i ) is a function used to find the sum of the i-th labeled image, and γ is a hyperparameter.

[0048] During the training phase, the final fused prediction results are supervised based on the loss function proposed above and the network model is guided to learn. During the test phase, we load the trained network model and make predictions on the image.

[0049] S4, fully connected conditional random field optimization: The predicted oil stain distribution image is optimized using the fully connected conditional random field algorithm combined with the original image. The fully connected conditional random field is a commonly used post-processing method in image segmentation. It mainly combines the relationship between pixels in the original image to further optimize the boundaries of the oil stain image output by the network and improve the misclassified areas, that is, to further improve the detected oil stain area and make the boundaries more accurate and clear.

[0050] S5, output the oil stain distribution binary map: the pixel points in the image whose pixel values ​​are greater than a certain threshold are considered to be oil stain points and their values ​​are set to 255, and the others are background points and their values ​​are set to 0.

[0051] The method of the present invention proposes a multi-level feature aggregation network as a solution for oil stain detection on electric trains. A cross-layer feature attention module (CLFA), a multi-level refined feature module (MRF), and a cross-layer feature enhancement module (CLFE) are proposed in the network model. The expression of enhanced features is clearer, the features are more representative, and the detection effect is better. In addition, in the multi-level feature aggregation network, a loss function is proposed, which combines weighted binary cross entropy loss and improved mean absolute value error loss to perform algorithm iteration and parameter adjustment and optimization on the multi-level feature aggregation network. The method can promote the network's learning of the oil stain area under the condition of extremely unbalanced distribution of the foreground area, i.e., the oil stain and background area, and can better detect oil stains under complex and changeable internal and external conditions of the electric train. It also has a good detection effect on oil stains of different shapes and sizes, and has good algorithm performance.

[0052] Example 2

[0053] This embodiment is similar to Figure 1The image distribution shown in the figure is used for training 756 images, including 502 images with oil stains and 254 normal images, and for testing 570 images with oil stains, which are collected by cameras. A method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network includes the following steps:

[0054] Step S1: Image acquisition and labeling: sort the images and divide them into normal images and images with oil stains, and use LabelMe software for manual labeling. That is, each image will generate a corresponding mask image, that is, the value of the pixel of the oil stain in the image is set to 255, and the pixel value of other parts is set to 0;

[0055] Step S2: Data preprocessing: The input original image and the corresponding mask image obtained in step S1 are preprocessed respectively. The preprocessing includes at least horizontal flipping, random rotation, cropping, random Gaussian blur image enhancement operations and normalization operations, that is, the corresponding channels of each image in the data set are subtracted by 0.485, 0.456, and 0.406 respectively and divided by 0.229, 0.224, and 0.225 respectively.

[0056] Step S3: Multi-level feature aggregation: The original image preprocessed in Step S2 and its corresponding mask image are input into a multi-level feature aggregation network to predict an image of the oil stain distribution. The difference between the predicted image and the mask image data annotated with the original image is calculated based on a loss function. The network parameters are continuously adjusted using the stochastic gradient descent algorithm, and the network weights are continuously updated. Specifically, the loss function is calculated on randomly selected samples each time, and the gradient is used to update the network parameters to obtain the most appropriate multi-level feature aggregation network model. The hyperparameters α and β in the loss function are set to 2 and 1, respectively, and γ is set to 2. The network's initial learning rate is set to 0.01, which gradually decays based on the number of iterations. The network parameters are optimized using the SGD algorithm. The input image resolution is set to 512*512, the total number of training iterations is 10,000, and the batch size is set to 2.

[0057] Step S4: Fully connected conditional random field optimization: The predicted image of the oil stain distribution is optimized using the fully connected conditional random field algorithm combined with the original image, that is, the oil stain distribution image is optimized based on the densecrf function in the pydensecrf library.

[0058] Step S5: Output the oil stain distribution binary map: the pixel points in the image with a pixel value greater than 0 are considered to be oil stain points and their values ​​are set to 255, and the others are background pixels with a value of 0.

[0059] After the detection of this embodiment, the oil stain position of the motor vehicle can be located, and some images are as follows: Figure 5 As shown, Figure 5 (a) is a schematic diagram of oil stain areas of various sizes. Figure 5 (b) is a schematic diagram of the detection of multiple oil stain areas in one image, providing an effective and appropriate method for locating the oil stain position in the train.

[0060] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A motor vehicle oil stain detection method based on a multi-level feature aggregation network, characterized in that: The method comprises the following steps: S1, image acquisition and annotation: obtain the original image based on the internal structure of the train, organize the original image, divide it into normal images and images with oil stains, and use LabelMe software for manual annotation, that is, each image will generate a corresponding mask image; S2, data preprocessing: preprocessing the input original image and the corresponding mask image obtained in step S1 respectively, wherein the preprocessing includes at least image enhancement and normalization operations; S3, multi-level feature aggregation: the original image pre-processed in step S2 and its corresponding mask image are input into the multi-level feature aggregation network to extract features, multi-level enhanced features and fused features, and predict the image of the oil stain distribution; based on the loss function, the gap between the distribution map and the mask image data annotated by the original image is calculated, and the network parameters are continuously adjusted based on the stochastic gradient descent algorithm, and the network weights are continuously updated, that is, each time a sample is randomly selected to calculate the loss function, the gradient is obtained to update the network parameters, and the most suitable multi-level feature aggregation network model is obtained; the multi-level feature aggregation network model at least includes a cross-layer feature attention module, a multi-level refined feature module, and a cross-layer feature enhancement module; wherein, The cross-layer feature attention module mainly transforms the features of multiple levels to the same scale based on convolution, fuses them by adding them, calculates the corresponding weights and multiplies them to the corresponding features, and finally adds them to the original features; The input of the multi-level feature refinement module is the upsampling result of the low-level large-scale features and the high-level small-scale features, or the splicing of multiple upsampling results. Features are extracted from different convolution methods, the results are added to the low-level features, and then enhanced based on different convolution methods. Finally, they are spliced ​​together and dimensionality reduction is performed based on convolution. The cross-layer feature enhancement module enhances low-level features based on high-level features, upsampling the features of the three higher levels to the same scale as the lowest-level features, concatenating them and feeding them into the MRF module for enhancement; S4, fully connected conditional random field optimization: The predicted image of the oil stain distribution is optimized using a fully connected conditional random field algorithm combined with the original image. Specifically, the optimization further optimizes the boundaries of the oil stain image output by the network and improves the misclassified areas based on the relationship between pixels in the original image; S5, oil trace distribution binary map output: After the optimization of step S4, the oil trace distribution binary map is obtained.

2. The method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network according to claim 1, characterized in that: In step S1, the pixel values ​​at the positions on the mask image corresponding to the oil stain areas of the original image are set to 255, and the values ​​of the remaining normal parts are set to 0.

3. The method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network according to claim 2, characterized in that: The image enhancement in step S2 is performed by randomly flipping the original image and its corresponding mask image horizontally to complete the data enhancement operation; the normalization operation is specifically as follows: based on the mean and standard deviation statistics on the ImageNet dataset, the three channels of the original image data are subtracted from the mean and divided by the standard deviation, and the mask image data is divided by 255 to normalize to 0-1.

4. The method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network according to claim 3, wherein: The loss function in step S3 is: L=α·L bce +β·L m Among them, α and β are the hyperparameters of adjustment; L bce is the binary cross entropy loss, L m is the improved mean absolute loss.

5. The method for detecting oil stains on motor vehicles based on a multi-level feature aggregation network according to claim 4, characterized in that: The improved mean absolute value loss L in step S3 m The specific expressions are as follows: Where M is the number of images in the training set; σ is the sigmoid function; p i is the i-th predicted image; g i is the i-th labeled image; sum(g i ) is a function used to find the sum of the i-th labeled image; γ is a hyperparameter.

Images