A rail locomotive surface defect detection method and system

Abstract

The application provides a rail locomotive surface defect detection method and system, comprising: obtaining a standard image and a defect image of an outer surface image; cutting the standard image to obtain a defect-free sample set; performing image correction and cutting on the defect image to obtain a defect sample set; training a twin CNN network through a loss function to obtain a trained twin CNN network; inputting the defect-free sample set into the trained twin CNN network to obtain a first feature tensor F1; inputting the defect sample set into the trained twin CNN network to obtain a second feature tensor F2; calculating the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain a distance map; and optimizing the distance map to obtain a final change map. The application constructs and trains a twin CNN network, which can be used for feature extraction of standard images and defect images, and solves the problem that image defect region features cannot be effectively extracted due to complex image backgrounds.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for detecting surface defects on rail locomotives. Background Technology

[0002] Because rail locomotives are large and complex transportation equipment, there are many types of defects that need to be detected, and the detection is difficult. Therefore, the current method of detecting defects in rail vehicles is still mainly the traditional method of manual visual inspection. However, this method has many problems: 1. High labor intensity for workers, poor detection stability and consistency; 2. High risk factor on both sides of the track, posing safety hazards to workers; 3. Low degree of automation and low detection efficiency; 4. High rate of missed detection and low level of detection precision; 5. Difficulty in recruiting and employing workers, and high cost.

[0003] Some research institutions have attempted to use machine vision and traditional image processing methods to detect surface defects on rail vehicles. This method involves acquiring images of the rail vehicle's exterior surface using appropriate light sources and image sensors (line scan cameras), extracting feature information from the images using corresponding image processing algorithms, and then locating and identifying surface defects based on this feature information. However, current methods for detecting surface defects on rail vehicles based on machine vision and traditional image processing suffer from the following problems: 1. They are affected by multiple factors such as environment, lighting, and light reflection from the vehicle surface, requiring a generally low signal-to-noise ratio in the detection system, making it difficult to detect weak defect signals or effectively distinguish them from noise; 2. The accuracy of defect detection methods based on machine vision and traditional image processing is not high. When the detection objects are diverse, the types of surface defects are numerous, their shapes and sizes vary, and the background is complex, extracting defect features is difficult. The compatibility of traditional image algorithms is insufficient, making it difficult to form an effective and unified defect detection method; 3. When dealing with large amounts of data, the computational complexity of the algorithm is too high, resulting in slow operation and poor real-time performance. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides a method for detecting surface defects on rail locomotives, comprising:

[0005] S1: Acquire images of the outer surface of the railcar, and obtain standard and defect images of the outer surface;

[0006] S2: Cropping the standard image to obtain a defect-free sample set; performing image correction and cropping on the defective image to obtain a defective sample set;

[0007] S3: Construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network;

[0008] S4: Input the set of undefective samples into the trained Siamese CNN network to obtain the first feature tensor F1; input the set of defective samples into the trained Siamese CNN network to obtain the second feature tensor F2;

[0009] S5: Calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map;

[0010] S6: Optimize the distance map to obtain the final change map, and obtain the defect area on the outer surface of the rail locomotive through the final change map.

[0011] Preferably, step S2 specifically includes:

[0012] S21: The standard image has a width of W1 and a height of H1. Obtain the distorted image from the defect image. The distorted image has a width of W2 and a height of H1. Create a first sliding window with a width of W0 and a second sliding window with a height of H1 on the standard image and the distorted image, respectively.

[0013] S22: Using the SIFT feature matching algorithm, feature points are matched between the region of the distorted image within the second sliding window and the region of the standard image within the first sliding window to obtain key points;

[0014] S23: Calculate the perspective transformation matrix of the distorted image within the second sliding window by matching key points to obtain the corrected image, and use the corrected image as the defect sample obtained by cropping the position of the second sliding window;

[0015] S24: Transfer the position of the second sliding window to the first sliding window so that the two sliding windows are in the same position in the pixel coordinate system, and use the standard image in the first sliding window as the defect-free sample obtained by cropping the position of the first sliding window.

[0016] S25: After moving the position of the second sliding window, repeat steps S21-S24 until the number of untraversed pixel columns is less than W0 and discard the untraversed areas. All the defect-free samples obtained are used as the defect-free sample set, and all the defective samples obtained are used as the defective sample set.

[0017] Preferably, the twin CNN network includes: a first branch network and a second branch network;

[0018] Both the first and second branch networks consist of five convolutional blocks;

[0019] Each convolutional block includes: a convolutional layer, a ReLU activation layer, and a normalization layer;

[0020] The kernel size of the convolutional layer in the last convolutional block is set to 1*1.

[0021] The kernel sizes of the convolutional layers in the first convolutional block to the fourth convolutional block are set to 3*3, 3*3, 5*5, and 5*5, respectively.

[0022] Preferably, the training process of the twin CNN network is as follows:

[0023] The network weight parameters of the Siamese CNN are updated using the loss function until the network loss converges. The calculation formula is as follows:

[0024]

[0025] Where Loss is the network loss; D i,j The distance is Euclidean; i and j are the x and y coordinates in the image pixel coordinate system; k is the order of training sample pairs; w u and w c These represent the weights of the changed and unchanged pixel pairs, respectively; y i，j y is the pixel variation coefficient, which is used when corresponding pixel pairs in the two images remain unchanged. i，j =0, if y changes i，j =1.

[0026] Preferably, the larger the Euclidean distance between F1 and F2 in the distance diagram, the greater the possibility that there is a difference between the corresponding pixels in the defective sample and the defective sample.

[0027] Preferably, step S6 specifically includes:

[0028] S61: Perform threshold segmentation on the distance map to obtain the initial change map;

[0029] S62: Improve the initial change map using the KNN nearest neighbor algorithm to obtain the final change map;

[0030] S63: The area with a grayscale value of 255 on the final change map is the defect area on the outer surface of the rail locomotive.

[0031] A surface defect detection system for rail locomotives, comprising:

[0032] The image acquisition module is used to acquire images of the outer surface of the rail locomotive, including standard and defect images of the outer surface.

[0033] The sample acquisition module is used to crop standard images to obtain a defect-free sample set; and to perform image correction and cropping on defective images to obtain a defective sample set.

[0034] The network training module is used to construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network.

[0035] The feature tensor acquisition module is used to input the defect-free sample set into the trained Siamese CNN network to obtain the first feature tensor F1; and to input the defective sample set into the trained Siamese CNN network to obtain the second feature tensor F2.

[0036] The distance map acquisition module is used to calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map;

[0037] The defect area acquisition module is used to optimize the distance map to obtain the final change map, and the defect area on the outer surface of the rail locomotive is obtained through the final change map.

[0038] A storage device that stores instructions and data for implementing the aforementioned method for detecting surface defects on rail locomotives.

[0039] A surface defect detection device for rail locomotives includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the surface defect detection method for rail locomotives.

[0040] The present invention has the following beneficial effects:

[0041] 1. This invention utilizes SIFT feature matching for track vehicle image correction. It can perform image correction by matching feature points between standard and distorted images, solving the image distortion problem caused by uneven track vehicle speed, simplifying the image correction process, and shortening the detection cycle.

[0042] 2. This invention constructs and trains a Siamese CNN network, which can be used to extract features from standard images and defective images. It solves the problem that the features of defective regions in images cannot be effectively extracted due to complex backgrounds, reduces the false negative rate and the false positive rate, and significantly improves the robustness of the algorithm. Attached Figure Description

[0043] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0044] Figure 2 This is a structural diagram of a twin CNN network;

[0045] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] Reference Figure 1 This invention provides a method for detecting surface defects on rail locomotives, comprising:

[0048] S1: Acquire images of the outer surface of the railcar, and obtain standard and defect images of the outer surface;

[0049] S2: Cropping the standard image to obtain a defect-free sample set; performing image correction and cropping on the defective image to obtain a defective sample set;

[0050] S3: Construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network;

[0051] S4: Input the set of undefective samples into the trained Siamese CNN network to obtain the first feature tensor F1; input the set of defective samples into the trained Siamese CNN network to obtain the second feature tensor F2;

[0052] S5: Calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map;

[0053] S6: Optimize the distance map to obtain the final change map, and obtain the defect area on the outer surface of the rail locomotive through the final change map.

[0054] Furthermore, step S1 specifically includes:

[0055] A hardware image acquisition platform consisting of high-resolution line array cameras and light sources fixed on both sides of the track is used. The host computer software for the cameras and light sources adjusts appropriate parameters such as exposure time, acquisition frequency, and light intensity to acquire images of the outer surface of the track locomotive for subsequent image processing and stores them in the computer.

[0056] Furthermore, in step S2, the distortion of the outer surface image of the railcar is corrected and cropped into multiple images at the same time. The simultaneous image correction and cropping saves the algorithm's running time, improves work efficiency, and enables the fully convolutional twin neural network to extract image features more accurately.

[0057] Step S2 is as follows:

[0058] S21: The standard image has a width of W1 and a height of H1. Obtain the distorted image from the defect image. The distorted image has a width of W2 and a height of H1. Create a first sliding window with a width of W0 and a second sliding window with a height of H1 on the standard image and the distorted image, respectively.

[0059] S22: Using the SIFT feature matching algorithm, feature points are matched between the region of the distorted image within the second sliding window and the region of the standard image within the first sliding window to obtain key points;

[0060] Specifically, traditional line scan camera images are usually calibrated by taking multiple images of the calibration board at different angles to correct image distortion. Although this method has high correction accuracy, it is cumbersome to operate and requires the calibration board to be placed in a specific location, which is not suitable for use on both sides of a railway. The image correction method based on SIFT feature matching can correct the image by using only the feature points of the image itself, avoiding the problem of not being able to place the calibration board, simplifying the image correction process and shortening the detection cycle.

[0061] The SIFT feature matching algorithm is used to downsample the image within the sliding window to obtain image pyramids at different resolutions. Then, Gaussian convolution is performed on each layer of the image to obtain the multi-scale space of the LOG image. Then, the LOG image is subtracted from its neighboring images to obtain the Gaussian difference pyramid reconstructed from all images. The extreme points of the feature map in the Gaussian difference pyramid are the key points of the image. Equation (1) is the expression of the two-dimensional Gaussian space convolution function, where Δx and Δ represent the horizontal and vertical coordinate offsets of the convolution kernel, respectively, and σ is the Gaussian kernel parameter. Equation (2) is the multi-scale space expression, where I(x, y) represents the convolved image.

[0062]

[0063] L(x,y,σ)=G(x,y,σ)*I(x,y) (2)

[0064] S23: Calculate the perspective transformation matrix of the distorted image in the second sliding window by matching key points to obtain the corrected image (at this time, the size of the corrected image is no longer consistent with the size of the sliding window), and use the corrected image as the defect sample obtained by cropping the position of the second sliding window;

[0065] S24: Transfer the position of the second sliding window to the first sliding window so that the two sliding windows are in the same position in the pixel coordinate system, and use the standard image in the first sliding window as the defect-free sample obtained by cropping the position of the first sliding window.

[0066] S25: After moving the position of the second sliding window, repeat steps S21-S24 until the number of remaining untraversed pixel columns is less than W0 and discard the untraversed area (since the end of the acquired image is not a pixel belonging to the rail locomotive part, discarding the untraversed pixels does not affect the feature extraction of the subsequent neural network). All the defect-free samples obtained are used as the defect-free sample set, and all the defective samples obtained are used as the defective sample set.

[0067] Furthermore, a Siamese CNN network (Siamese Fully Convolutional Neural Network FSCNN) is constructed for image feature extraction. Traditional image feature extraction techniques generally employ manually designed methods, which rely on databases and require design based on the characteristics of the data. However, the image feature extraction method based on Siamese CNN can overcome the shortcomings of traditional methods. Through convolutional layers and weight sharing, it can effectively extract defect information in the image, significantly improving the robustness of the algorithm.

[0068] The Siamese CNN network takes a flawless standard image and a defective image as input and outputs two feature tensors. The input and output dimensions are identical, and the two branches of the network share the same weights, ensuring that the methods for extracting features from the two images are the same. The structure of the Siamese CNN network is as follows: Figure 2 As shown;

[0069] The twin CNN network includes: a first branch network and a second branch network;

[0070] Both the first and second branch networks consist of five convolutional blocks;

[0071] Each convolutional block includes: a convolutional layer, a ReLU activation layer, and a normalization layer;

[0072] The kernel size of the convolutional layer in the last convolutional block is set to 1*1 (for feature selection);

[0073] The kernel sizes of the convolutional layers in the first convolutional block to the fourth convolutional block are set to 3*3, 3*3, 5*5 and 5*5 respectively (to ensure the expansion of the receptive field).

[0074] Furthermore, the training process of the twin CNN network is as follows:

[0075] The network weight parameters of the Siamese CNN are updated using the loss function until the network loss converges. The calculation formula is as follows:

[0076]

[0077] Where Loss is the network loss; D i,j The distance is Euclidean; i and j are the x and y coordinates in the image pixel coordinate system; k is the order of training sample pairs; w u and w c These represent the weights of the changed and unchanged pixel pairs, respectively; y i，j y is the pixel variation coefficient, which is used when corresponding pixel pairs in the two images remain unchanged. i，j =0, if y changes i，j =1.

[0078] Furthermore, the larger the Euclidean distance between F1 and F2 in the distance diagram, the greater the possibility that there is a difference between the corresponding pixels in the defective sample and the defective sample.

[0079] Specifically, in order to compare the degree of difference between the defect-free standard image and the defective image, the Euclidean distance method of calculating the two output tensors of the Siamese CNN network is used. The larger the result value, the more likely there is a difference between the corresponding pixels of the two input images; Equation (4) is the formula for calculating the Euclidean distance D, G(X1). i，j And G(X2) i，j These represent the two feature tensors output by the twin CNN network.

[0080] D(X1,X2)=||G(X1) i,j -G(X2) i,j ||2 (4)

[0081] Furthermore, step S6 specifically involves:

[0082] S61: Perform threshold segmentation on the distance map to obtain the initial change map;

[0083] Specifically, a variation diagram is an image that shows the areas of difference between a defective image and a standard image without defects;

[0084] S62: Improve the initial change map using the KNN nearest neighbor algorithm to obtain the final change map;

[0085] Specifically, the purpose of using the KNN nearest neighbor algorithm is to eliminate discrete noise regions on the initial change map, making the defect region determination more accurate;

[0086] S63: The area with a grayscale value of 255 on the final change map is the defect area on the outer surface of the rail locomotive.

[0087] This invention provides a surface defect detection system for rail locomotives, comprising:

[0088] The image acquisition module is used to acquire images of the outer surface of the rail locomotive, including standard and defect images of the outer surface.

[0089] The sample acquisition module is used to crop standard images to obtain a defect-free sample set; and to perform image correction and cropping on defective images to obtain a defective sample set.

[0090] The network training module is used to construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network.

[0091] The feature tensor acquisition module is used to input the defect-free sample set into the trained Siamese CNN network to obtain the first feature tensor F1; and to input the defective sample set into the trained Siamese CNN network to obtain the second feature tensor F2.

[0092] The distance map acquisition module is used to calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map;

[0093] The defect area acquisition module is used to optimize the distance map to obtain the final change map, and the defect area on the outer surface of the rail locomotive is obtained through the final change map.

[0094] The present invention provides a storage device that stores instructions and data for implementing the above-described method for detecting surface defects on rail locomotives.

[0095] This invention provides a surface defect detection device for rail locomotives, comprising: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the above-mentioned surface defect detection method for rail locomotives.

[0096] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0097] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0098] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting surface defects on a railcar, characterized in that, include: S1: Acquire images of the outer surface of the railcar, and obtain standard and defect images of the outer surface; S2: Cropping the standard image to obtain a defect-free sample set; Image correction and cropping are performed on defective images to obtain a defect sample set; S3: Construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network; S4: Input the defect-free sample set into the trained Siamese CNN network to obtain the first feature tensor F1; The defect sample set is input into the trained Siamese CNN network to obtain the second feature tensor F2; S5: Calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map; S6: Optimize the distance map to obtain the final change map, and use the final change map to obtain the defect area on the outer surface of the rail locomotive; Step S2 is as follows: S21: The standard image has a width of W1 and a height of H1. Obtain the distorted image from the defect image. The distorted image has a width of W2 and a height of H1. Create a first sliding window with a width of W0 and a second sliding window with a height of H1 on the standard image and the distorted image, respectively. S22: Using the SIFT feature matching algorithm, feature points are matched between the region of the distorted image within the second sliding window and the region of the standard image within the first sliding window to obtain key points; S23: Calculate the perspective transformation matrix of the distorted image within the second sliding window by matching key points to obtain the corrected image, and use the corrected image as the defect sample obtained by cropping the position of the second sliding window; S24: Transfer the position of the second sliding window to the first sliding window so that the two sliding windows are in the same position in the pixel coordinate system, and use the standard image in the first sliding window as the defect-free sample obtained by cropping the position of the first sliding window. S25: After moving the position of the second sliding window, repeat steps S21-S24 until the number of untraversed pixel columns is less than W0 and discard the untraversed area. All the defect-free samples obtained are used as the defect-free sample set, and all the defective samples obtained are used as the defective sample set. The twin CNN network includes: a first branch network and a second branch network; Both the first and second branch networks consist of five convolutional blocks; Each convolutional block includes: a convolutional layer, a ReLU activation layer, and a normalization layer; The kernel size of the convolutional layer in the last convolutional block is set to 1*1. The kernel sizes of the convolutional layers in the first convolutional block to the fourth convolutional block are set to 3*3, 3*3, 5*5 and 5*5 respectively. The training process of the twin CNN network is as follows: The network weight parameters of the Siamese CNN are updated using the loss function until the network loss converges. The calculation formula is as follows: Where Loss is the network loss; D i,j The distance is Euclidean; i and j are the x and y coordinates in the image pixel coordinate system; k is the order of training sample pairs; w u and w c These represent the weights of the changed and unchanged pixel pairs, respectively; y i，j y is the pixel variation coefficient, which is used when corresponding pixel pairs in the two images remain unchanged. i，j =0, if y changes i，j =1.

2. The method for detecting surface defects on rail locomotives according to claim 1, characterized in that, The larger the Euclidean distance between F1 and F2 in the distance diagram, the greater the possibility that there is a difference between the corresponding pixels in the defective sample and the defective sample.

3. The method for detecting surface defects on rail locomotives according to claim 1, characterized in that, Step S6 is as follows: S61: Perform threshold segmentation on the distance map to obtain the initial change map; S62: Improve the initial change map using the KNN nearest neighbor algorithm to obtain the final change map; S63: The area with a grayscale value of 255 on the final change map is the defect area on the outer surface of the rail locomotive.

4. A surface defect detection system for rail locomotives, characterized in that, include: The image acquisition module is used to acquire images of the outer surface of the rail locomotive, including standard and defect images of the outer surface. The sample acquisition module is used to crop standard images to obtain a defect-free sample set; Image correction and cropping are performed on defective images to obtain a defect sample set; The network training module is used to construct a Siamese CNN network, train the Siamese CNN network using a loss function, and obtain a trained Siamese CNN network. The feature tensor acquisition module is used to input the defect-free sample set into the trained Siamese CNN network to obtain the first feature tensor F1. The defect sample set is input into the trained Siamese CNN network to obtain the second feature tensor F2; The distance map acquisition module is used to calculate the Euclidean distance between the first feature tensor F1 and the second feature tensor F2 to obtain the distance map; The defect area acquisition module is used to optimize the distance map to obtain the final change map, and to obtain the defect area on the outer surface of the rail locomotive through the final change map; The specific processing procedure of the sample acquisition module is as follows: The standard image has a width of W1 and a height of H1. The distorted image in the defect image is obtained. The distorted image has a width of W2 and a height of H1. A first sliding window with a width of W0 and a height of H1 are created on the standard image and the distorted image, respectively. By using the SIFT feature matching algorithm, the region of the distorted image within the second sliding window is matched with the region of the standard image within the first sliding window to obtain key points; The perspective transformation matrix of the distorted image within the second sliding window is calculated by matching key points to obtain the corrected image. The corrected image is then used as the defect sample obtained by cropping the second sliding window position. The position of the second sliding window is passed to the first sliding window so that the two sliding windows are in the same position in the pixel coordinate system. The standard image in the first sliding window is used as the defect-free sample obtained by cropping the position of the first sliding window. After moving the position of the second sliding window, repeat steps S21-S24 until the number of untraversed pixel columns is less than W0 and discard the untraversed areas. All the defect-free samples obtained are used as the defect-free sample set, and all the defective samples obtained are used as the defective sample set. The twin CNN network includes: a first branch network and a second branch network; Both the first and second branch networks consist of five convolutional blocks; Each convolutional block includes: a convolutional layer, a ReLU activation layer, and a normalization layer; The kernel size of the convolutional layer in the last convolutional block is set to 1*1. The kernel sizes of the convolutional layers in the first convolutional block to the fourth convolutional block are set to 3*3, 3*3, 5*5 and 5*5 respectively. The training process of the twin CNN network is as follows: The network weight parameters of the Siamese CNN are updated using the loss function until the network loss converges. The calculation formula is as follows: Where Loss is the network loss; D i,j The distance is Euclidean; i and j are the x and y coordinates in the image pixel coordinate system; k is the order of training sample pairs; w u and w c These represent the weights of the changed and unchanged pixel pairs, respectively; y i，j y is the pixel variation coefficient, which is used when corresponding pixel pairs in the two images remain unchanged. i，j =0, if y changes i，j =1.

5. A storage device, characterized in that: The storage device stores instructions and data for implementing the rail locomotive surface defect detection method according to any one of claims 1 to 3.

6. A surface defect detection device for rail locomotives, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the method for detecting surface defects of a rail locomotive as described in any one of claims 1 to 3.

Images