Bullet train front coupler cover loss fault image identification method based on deep learning

Abstract

The invention discloses a bullet train front coupler cover loss fault image recognition method based on deep learning, solves the problem of how to improve the front coupler cover loss fault detection accuracy, and belongs to the technical field of locomotive fault detection. The method comprises the steps of obtaining an image of a front car coupler cover, and establishing a sample data set; establishing a fault detection model, wherein the fault detection model comprises a ResNet50 network, a feature pyramid FPN, a region-of-interest generation network RPN and a detector; training a fault detection model by using the sample data set; and acquiring a bullet train image, detecting the front coupler cover in the image by using the trained fault detection model, and judging whether the front coupler cover is lost or not. According to the fault detection model based on deep learning, the feature pyramid FPN is improved, the feature fusion mode of the improved FPN is more flexible and diversified, information in the image is captured and utilized more comprehensively, the feature fusion and extraction capability can be remarkably improved, and the performance of the fault detection model can be remarkably improved.

Description

Technical Field

[0001] The invention relates to a method for recognizing a motor vehicle front coupler cover missing fault image based on deep learning, and belongs to the technical field of locomotive fault detection. Background Art

[0002] The front coupler cover missing fault is a fault that endangers driving safety. In the front coupler cover missing fault detection, the fault detection is performed by manually checking the image. Since the inspectors are very likely to be fatigued and miss things during work, missed detection and wrong detection will occur, affecting driving safety. The use of automatic image recognition can improve the detection efficiency and stability. In recent years, deep learning and artificial intelligence have continued to develop and mature in technology. Therefore, the use of deep learning for front coupler cover missing fault identification can effectively improve the detection accuracy. The detection accuracy of existing deep learning-based detection methods needs to be improved. Summary of the invention

[0003] In order to improve the detection accuracy of the front coupler cover missing fault, the present invention provides a method for image recognition of the front coupler cover missing fault of a motor vehicle based on deep learning.

[0004] A method for recognizing a fault image of a motor vehicle front coupler cover missing based on deep learning of the present invention comprises:

[0005] S1, obtaining an image of the front coupler cover and establishing a sample data set;

[0006] S2. Establish a fault detection model, including ResNet50 network, feature pyramid FPN, region of interest generation network RPN and detector;

[0007] The image is input to the ResNet50 network. The ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5. The convolution modules Conv1 to Conv5 extract features and input them to the feature pyramid FPN after downsampling. The feature pyramid FPN includes three layers of nodes. The first layer includes two nodes, and the second and third layers each include four nodes. Each node fuses the input and outputs it.

[0008] The output of the convolution module Conv2 is downsampled and input to the first node of the first layer and the first node of the second layer at the same time;

[0009] The output of the convolution module Conv3 is downsampled and input to the first node of the first layer and the second node of the second layer at the same time;

[0010] The output of the convolution module Conv4 is downsampled and input to the second node of the first layer and the third node of the second layer at the same time;

[0011] The output of the convolution module Conv5 is downsampled and input to the fourth node of the second layer;

[0012] The output of the first node of the first layer is input to the second node of the second layer after downsampling, and the output of the second node of the second layer is input to the fourth node of the second layer after downsampling;

[0013] At the same time, the outputs of the two nodes in the first layer are respectively input to the second and third nodes in the second layer, the four nodes in the second layer are connected from bottom to top, the four nodes in the third layer are connected from top to bottom, the outputs of the first three nodes in the third layer are recursively input to the inputs of the first three nodes in the second layer, the outputs of the four nodes in the second layer are respectively input to the inputs of the four nodes in the third layer, and the outputs of the four nodes in the third layer are simultaneously input to the region of interest generation network RPN;

[0014] The region of interest generation network RPN extracts the candidate frame image from the input feature map, the candidate frame image is input to the detector, and the detector outputs the detection result;

[0015] S3, using the sample data set to train the fault detection model;

[0016] S4. Obtain an image of a motor vehicle, and use the trained fault detection model to detect the front vehicle coupler cover in the image to determine whether it is lost.

[0017] Preferably, the ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5, and the convolution modules Conv2 to Conv5 output corresponding convolution layer features respectively.

[0018] Preferably, the ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5, and the activation function in each convolution module adopts the ELU function:

[0019]

[0020] Among them, x is the input data and a is a fixed parameter in the interval (1, +∞).

[0021] Preferably, the region generation network RPN uses a Gaussian non-maximum suppression method to extract candidate boxes.

[0022] Preferably, the detector includes three ROI pooling layers, a classification network C3, and a target position regression positioning network B0 to B3;

[0023] The candidate box image output by the region of interest generation network RPN is input into the regression positioning network B0 of the target position, and the output of the regression positioning network B0 of the target position is simultaneously input into three ROI pooling layers, the output of the first ROI pooling layer is input into the regression positioning network B1 of the target position, the output of the regression positioning network B1 of the target position is input into the second ROI pooling layer, the output of the second ROI pooling layer is input into the regression positioning network B2 of the target position, the output of the regression positioning network B2 of the target position is input into the third ROI pooling layer, and the output of the third ROI pooling layer is simultaneously input into the classification network C3 and the regression positioning network B3 of the target position.

[0024] Preferably, S1 includes: setting up high-definition equipment around the EMU track, and obtaining various high-definition images after the EMU passes through the high-definition equipment;

[0025] Based on various high-definition images, two types of images, one with a front coupler cover and one without a front coupler cover, are captured, amplified, and marked to complete the construction of the sample data set.

[0026] Preferably, the amplification includes translation, flipping, brightness adjustment and noise addition of the image.

[0027] The beneficial effects of the present invention are as follows: the feature pyramid FPN is improved in the fault detection model based on deep learning. The improved FPN fusion feature method is more flexible and diverse, and can more comprehensively capture and utilize the information in the image, which can significantly improve the feature fusion and extraction capabilities, and can significantly improve the performance of the fault detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart for fault identification;

[0029] Figure 2 This is the fault detection model of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0032] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0033] The deep learning-based image recognition method for a motor vehicle front coupler cover missing fault in this embodiment includes:

[0034] Step 1: Set up high-definition imaging equipment around the EMU tracks, obtain images of the front coupler cover after the EMU passes through the equipment, and establish a sample data set;

[0035] Step 2: Establish a fault detection model, including a backbone network, a feature pyramid FPN, a region of interest generation network RPN, and a detector; use a deep learning network to detect the front hook cover in the image.

[0036] Step 3: Use the sample data set to train the fault detection model;

[0037] Step 4: Get the train image and use the trained fault detection model to detect the front coupler cover in the image to determine whether it is missing. The missing front coupler cover will be uploaded and an alarm will be issued. The staff will take corresponding measures based on the recognition results to ensure the safe operation of the train.

[0038] In step 1, high-definition equipment is set up around the EMU track. After the EMU passes through the high-definition equipment, various high-definition images are obtained. Based on the various high-definition images, two images with and without the front coupler cover are intercepted, amplified, and marked to complete the construction of the sample data set:

[0039] Preprocessing, crop the image to a specified size. The image scaling of each manufacturer is different. Scale it to a rough proportion according to the positioning. Set up high-definition equipment around the EMU track. Get high-definition images after the EMU passes through the equipment. The image is a clear grayscale image. Since EMU parts may be affected by natural conditions or artificial conditions such as rain, mud stains, oil stains, black paint, etc., and there may be differences in images taken at different sites. Therefore, the images of the front hook cover vary greatly. Therefore, in the process of collecting the image data of the front hook cover, it is necessary to ensure diversity and try to collect all the images of the front hook cover under various conditions.

[0040] The shapes of the front hook covers may be different in different types of vehicle models. All types of front hook covers are collectively referred to as one category, and a sample data set is established based on the category.

[0041] The sample data set is mainly divided into two parts, with and without the front coupler cover, both of which are grayscale images. The screenshot of the front coupler cover includes: grayscale image set and labeled image set. The grayscale image set is a high-definition grayscale image taken by the device. The labeled image set is the segmented image of the front coupler cover. This part of the image is a grayscale image obtained by manual labeling. There is a one-to-one correspondence between the grayscale image data set and the labeled image data set, that is, each grayscale image corresponds to a labeled image.

[0042] Although the sample data set includes images under various conditions, it is still necessary to perform data augmentation on the sample data set to improve the stability of the algorithm. Augmentation includes operations such as image translation, flipping, brightness adjustment, and adding noise. Each operation is performed under random conditions, which can maximize the diversity and applicability of the samples.

[0043] The fault detection model in step 2 is an improved Cascade R-cnn network structure. The fault detection model of this embodiment includes a backbone network, a feature pyramid FPN, a region of interest generation network RPN, and a detector composed of multiple classifiers and regressors. The backbone network uses a ResNet50 network, such as Figure 2 As shown, Conv1~Conv5 represent ResNet50 convolution modules, and each convolution module contains multiple convolution layers. This embodiment adopts ELU as the feature extraction network activation function, as shown in formula (1), where a is a fixed parameter in the interval (1, +∞). The original activation function uses Relu, as shown in formula (2). When the input parameter of Relu is negative, the output is zero and its derivative is also zero, resulting in the inability to update the parameters when the network training gradient is reversely conducted, which reduces the learning ability of the network to a certain extent. When the input parameter of the ELU used in this patent is negative, the output of the network is not zero, and its derivative is not zero. When the network training gradient is reversely conducted, the parameters can be updated normally, thereby improving the learning ability of the network.

[0044]

[0045]

[0046] In order to make full use of the shallow features of the feature extraction network, improve the feature extraction capability of the network, and enhance the performance of the detection network, this implementation adopts an improved feature pyramid to optimize Resnet50 and realize the fusion of feature maps of different convolutional layers of Resnet50, such as Figure 2 The improved FPN has a more flexible and diverse way of fusion features, which can capture and utilize the information in the image more comprehensively, significantly improve the fusion and extraction capabilities of features, and significantly improve the performance of the detection network. The improved FPN includes three layers of nodes, the first layer includes two nodes, the second layer and the third layer each include four nodes, and each node fuses the input and outputs it;

[0047] The output of the convolution module Conv2 is downsampled and input to the first node of the first layer and the first node of the second layer at the same time;

[0048] The output of the convolution module Conv3 is downsampled and input to the first node of the first layer and the second node of the second layer at the same time;

[0049] The output of the convolution module Conv4 is downsampled and input to the second node of the first layer and the third node of the second layer at the same time;

[0050] The output of the convolution module Conv5 is downsampled and input to the fourth node of the second layer;

[0051] The output of the first node of the first layer is input to the second node of the second layer after downsampling, and the output of the second node of the second layer is input to the fourth node of the second layer after downsampling;

[0052] At the same time, the outputs of the two nodes in the first layer are respectively input to the second and third nodes in the second layer, the four nodes in the second layer are connected from bottom to top, the four nodes in the third layer are connected from top to bottom, the outputs of the first three nodes in the third layer are recursively input to the inputs of the first three nodes in the second layer, the outputs of the four nodes in the second layer are respectively input to the inputs of the four nodes in the third layer, and the outputs of the four nodes in the third layer are simultaneously input to the region of interest generation network RPN;

[0053] The improved FPN in this implementation mainly improves the feature fusion part, such as Figure 2 As shown in the figure, the input image is firstly subjected to feature extraction by the ResNet50 residual network, with a convolution size of 7×7, a sliding step of 2, a padding of 3, and an output channel number of 64. Then, it is preprocessed by 3×3 maximum pooling, and the convolution kernels are set to 1×1, 3×3, and 1×1 in the first layer for convolution output, and the output channel number is 256. The 2nd, 3rd, and 4th layers perform similar convolution operations to extract features. In this process, the resolution of the feature map is continuously reduced, and the number of channels is continuously increased. Finally, the number of channels is adjusted to 256 through 1×1 convolution to generate feature maps.

[0054] a. The first layer of FPN (the leftmost layer), the feature map is adjusted in the same way as Cascade-rcnn, which is top-down and horizontally connected. If a node has only one input and there is no feature fusion, then its contribution to the overall feature network is relatively small and it can be removed.

[0055] b. The second layer (middle layer) of FPN is fused with the feature map of the first layer or directly with the previous layer after upsampling, so that more functions can be integrated without increasing the cost.

[0056] c. In the third layer of FPN (the rightmost layer), the feature map of the bottom-up path inherits the rich semantic information and accurate positioning information of the top-down path. In this process, the number of subsampling is less and the information loss is less. The feature map with smaller resolution uses the nearest neighbor interpolation algorithm to make the spatial resolution twice that of the original feature map, and the generated feature map is fused with the top-down feature map. The fused feature map is recursively sent to the second layer. Finally, the fused feature map is again eliminated through 3×3 convolution to generate the final feature map.

[0057] The region of interest generation network RPN extracts candidate box images from the input feature map. The non-maximum suppression NMS (non-maximum suppression) method is based on the category score, so only the candidate box with the highest score can be retained. However, in general, IoU is not strongly correlated with the category score. Therefore, there are deviations in the positions of many inspection boxes with high confidence levels for classification labels. For multiple overlapping targets, the detection algorithm should output multiple target detection boxes, and NMS will first select the candidate box with the highest confidence level, and then directly set the scores of all candidate boxes with IoU greater than the threshold to zero, resulting in only one target being detected, causing missed detections.

[0058] In this embodiment, the region of interest generation network RPN uses the Gaussian non-maximum suppression method Soft-NMS to replace NMS to solve the above problem. For the box whose IoU with the highest scoring candidate box is greater than the threshold, it is not deleted directly, but suppressed by reducing the confidence. The greater the IoU value with the highest scoring box, the greater the degree of suppression. Using the Gaussian non-maximum suppression method Soft-NMS to replace NMS can obtain more accurate candidate boxes.

[0059] The detector of this embodiment uses a cascaded classification and regression network to detect the feature map generated by the feature extraction network, such as Figure 2The right part. C1~C3 represents the classification network, whose output is the category of the detected target, B0~B3 is the regression positioning network of the target position, whose output is the position of the target in the image, and the output of the improved fault detection model is C3 and B3. The detector includes three ROI pooling layers, the classification network C3, and the regression positioning network B0~B3 of the target position; the candidate box image output by the region of interest generation network RPN is input to the regression positioning network B0 of the target position, and the output of the regression positioning network B0 of the target position is simultaneously input to the three ROI pooling layers, the output of the first ROI pooling layer is input to the regression positioning network B1 of the target position, the output of the regression positioning network B1 of the target position is input to the second ROI pooling layer, the output of the second ROI pooling layer is input to the regression positioning network B2 of the target position, the output of the regression positioning network B2 of the target position is input to the third ROI pooling layer, and the output of the third ROI pooling layer is simultaneously input to the classification network C3 and the regression positioning network B3 of the target position. The output feature map of each layer of the improved FPN contains the shallow and deep features of the feature extraction network Resnet50. The detector uses cascade operation with the improved FPN to improve the accuracy of classification and positioning, and improves the detection performance of the detection network. The fault detection model performs RPN optimization on the output features of each layer of the improved FPN and then performs ROI pooling to classify and locate the target.

[0060] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A method for image recognition of a motor vehicle front coupler cover missing fault based on deep learning, characterized in that: The method comprises: S1, obtaining an image of the front coupler cover and establishing a sample data set; S2. Establish a fault detection model, including ResNet50 network, feature pyramid FPN, region of interest generation network RPN and detector; The image is input to the ResNet50 network. The ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5. The convolution modules Conv1 to Conv5 extract features and input them to the feature pyramid FPN after downsampling. The feature pyramid FPN includes three layers of nodes. The first layer includes two nodes, and the second and third layers each include four nodes. Each node fuses the input and outputs it. The output of the convolution module Conv2 is downsampled and input to the first node of the first layer and the first node of the second layer at the same time; The output of the convolution module Conv3 is downsampled and input to the first node of the first layer and the second node of the second layer at the same time; The output of the convolution module Conv4 is downsampled and input to the second node of the first layer and the third node of the second layer at the same time; The output of the convolution module Conv5 is downsampled and input to the fourth node of the second layer; The output of the first node of the first layer is input to the second node of the second layer after downsampling, and the output of the second node of the second layer is input to the fourth node of the second layer after downsampling; At the same time, the outputs of the two nodes in the first layer are respectively input to the second and third nodes in the second layer, the four nodes in the second layer are connected from bottom to top, the four nodes in the third layer are connected from top to bottom, the outputs of the first three nodes in the third layer are recursively input to the inputs of the first three nodes in the second layer, the outputs of the four nodes in the second layer are respectively input to the inputs of the four nodes in the third layer, and the outputs of the four nodes in the third layer are simultaneously input to the region of interest generation network RPN; The region of interest generation network RPN extracts the candidate frame image from the input feature map, the candidate frame image is input to the detector, and the detector outputs the detection result; S3, using the sample data set to train the fault detection model; S4. Obtain an image of a motor vehicle, and use the trained fault detection model to detect the front vehicle coupler cover in the image to determine whether it is lost.

2. The deep learning-based image recognition method for the front coupler cover missing fault of a motor vehicle according to claim 1 is characterized in that: The ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5, and the convolution modules Conv2 to Conv5 output corresponding convolution layer features respectively.

3. The deep learning-based image recognition method for the front coupler cover missing fault of a motor vehicle according to claim 2 is characterized in that: The ResNet50 network includes sequentially connected convolution modules Conv1 to Conv5, and the activation function in each convolution module adopts the ELU function: Among them, x is the input data and a is a fixed parameter in the interval (1, +∞).

4. The deep learning-based image recognition method for a motor vehicle front coupler cover missing fault according to claim 1 is characterized in that: The region generation network RPN uses a Gaussian non-maximum suppression method to extract candidate boxes.

5. The deep learning-based image recognition method for a motor vehicle front coupler cover missing fault according to claim 1 is characterized in that: The detector includes three ROI pooling layers, a classification network C3, and target position regression and positioning networks B0 to B3; The candidate box image output by the region of interest generation network RPN is input into the regression positioning network B0 of the target position, and the output of the regression positioning network B0 of the target position is simultaneously input into three ROI pooling layers, the output of the first ROI pooling layer is input into the regression positioning network B1 of the target position, the output of the regression positioning network B1 of the target position is input into the second ROI pooling layer, the output of the second ROI pooling layer is input into the regression positioning network B2 of the target position, the output of the regression positioning network B2 of the target position is input into the third ROI pooling layer, and the output of the third ROI pooling layer is simultaneously input into the classification network C3 and the regression positioning network B3 of the target position.

6. The deep learning-based image recognition method for a motor vehicle front coupler cover missing fault according to claim 1 is characterized in that: S1 includes: Set up high-definition equipment around the EMU tracks, and obtain a variety of high-definition images when the EMU passes through the high-definition equipment; Based on various high-definition images, two types of images, one with a front coupler cover and one without a front coupler cover, are captured, amplified, and marked to complete the construction of the sample data set.

7. The deep learning-based image recognition method for a motor vehicle front coupler cover missing fault according to claim 6 is characterized in that: The amplification includes image translation, flipping, brightness adjustment and noise addition.

8. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the method for identifying the fault image of the missing front coupler cover of a motor vehicle based on deep learning are implemented as described in any one of claims 1 to 7.

9. A deep learning-based image recognition device for a motor vehicle front coupler cover missing fault, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of the deep learning-based image recognition method for the missing front coupler cover fault of a motor vehicle as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for identifying the fault image of the missing front coupler cover of a motor vehicle based on deep learning are implemented as described in any one of claims 1 to 7.

Images