A method for detecting missing bolts on train bottom based on image processing

Abstract

The present invention discloses a method for detecting missing bolts under a train vehicle based on image processing. The method comprises the following steps: acquiring original images of the train vehicle underbody using multiple linear array cameras, and obtaining the vehicle speed using two sets of dual-magnetic steel speed measurement methods; performing speed curve fitting to correct the distortion of the original images; performing illumination unevenness correction on the distortion-corrected images to obtain an illumination-corrected underbody image; combining the FAST and SURF algorithms to extract and describe feature points from the illumination-corrected underbody image; combining an improved k-d tree nearest neighbor search algorithm with a two-way matching adaptive threshold method to stitch the illumination-corrected underbody images collected by different cameras to obtain a complete underbody image; and performing data set creation and model training on the complete underbody image set. Finally, a deep learning target detection (SSD) algorithm is used to detect missing bolts under the vehicle underbody. The method has strong real-time performance and high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and target detection, in particular to an image processing-based method for detecting missing bolts on a train bottom. Background Art

[0002] For subways, the underbody of subway trains is the most complex and has the most intricate parts, so ensuring the safety of the underbody is crucial. Subway train underbody inspections primarily involve inspections at depots and factories, which are costly, prone to missed inspections, and employ relatively outdated methods. Manual inspections present numerous problems, relying primarily on the most primitive methods of train inspectors, such as visual inspection, hand contact, and perhaps the use of tools like hammers. These methods are not only inefficient but also burdensome for train inspectors. In general, this inspection method is completely dependent on the train inspector's condition, which can pose safety risks when the train inspector's workload is excessive and their condition is poor.

[0003] Bolts are small, yet essential components. Currently, bolt inspection relies primarily on the train inspector's experience and condition, which is highly subjective. Furthermore, bolts are numerous and diverse in shape, making inspection challenging. Furthermore, bolt inspection requires real-time performance. In recent years, with the rapid advancement of science, image recognition technology has become increasingly mature, making it suitable for undercarriage fault detection. Area scan cameras are traditionally used to capture undercarriage images. When a train passes, the acquisition system triggers the camera to take a photo. However, using area scan cameras presents several challenges. For example, due to the inherent nature of area scan cameras, the frequency of undercarriage image acquisition remains constant, while train speed fluctuates. Consequently, when the acquisition frequency and speed do not align, image distortion, such as compression and stretching, can occur. Furthermore, subway trains are relatively long, making it difficult for area scan cameras to capture the entire undercarriage image in a single acquisition. Consequently, multiple acquisitions are required, resulting in low acquisition efficiency.

[0004] Currently, the main image detection systems developed in China for different vehicle models are TFDS and TVDS. In today's TFDS system, the relatively low pixel count of the hardware hinders fault detection. Furthermore, fault determination requires manual evaluation of captured images, which can result in missed detections. Existing TEDS systems suffer from significant image misalignment and poor image registration, resulting in suboptimal identification of suspected faults. TVDS also faces challenges such as suboptimal image stitching, uneven image brightness, and suboptimal image quality under unusual circumstances. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic detection method for missing train bottom bolts with strong real-time performance and high detection accuracy, which can realize online dynamic detection of missing train bottom bolts.

[0006] The technical solution to achieve the purpose of the present invention is: a method for detecting missing bolts on the underbody of a train based on image processing, the steps of which are as follows:

[0007] Step 1: Capturing the original image of the train bottom: Multiple linear array cameras are used to obtain the original image of the train bottom, while two sets of dual-magnetic steel speed measurement methods are used to obtain the train speed;

[0008] Step 2: Distortion correction of the original vehicle bottom image: The original vehicle bottom image is distorted by compression or stretching due to the mismatch between the vehicle speed and the camera scanning rate. A speed curve is fitted based on the acquired vehicle speed data using arc curve fitting technology. The original vehicle bottom image is then subjected to distortion correction by stretching or compressing the fitted curve.

[0009] Step 3: Correcting uneven illumination: Using a two-dimensional Gamma function adaptive correction algorithm, perform uneven illumination correction on the distortion-corrected image to obtain a vehicle bottom image after uneven illumination correction.

[0010] Step 4: Stitching images from multiple camera modules: The FAST and SURF algorithms are combined to extract and describe feature points from the vehicle bottom image after uneven illumination correction. Then, an improved kd-tree nearest neighbor search algorithm (BBF) is combined with an adaptive threshold method for bidirectional matching to stitch the vehicle bottom images collected by different cameras after uneven illumination correction to obtain a complete image of the vehicle bottom.

[0011] Step 5: Bolt loss detection: Create a dataset and train a model based on the acquired complete underbody image set, and use the deep learning target detection algorithm (SSD) to detect missing bolts on the underbody.

[0012] Compared with the prior art, the present invention has the following significant advantages: (1) a linear array camera with an internal integrated compensation light source is used for image acquisition. The linear array camera is a linear scan, so the lateral resolution is high, and the trigger frequency can be changed during the acquisition process, which can also reduce distortion, and has the advantages of high clarity and high precision; (2) due to the limitations of on-site hardware equipment installation conditions, the compensation light source does not have good installation conditions, and the ET series line scanning module product solves this problem well. The module integrates a linear array camera and a high-power light source; (3) the image is corrected for distortion by using vehicle speed curve fitting, which solves the image distortion that occurs during the image acquisition process, and the image is corrected for uneven illumination, which solves the image quality problem caused by uneven light source compensation during the image acquisition process; (4) a target detection algorithm with higher detection accuracy for small targets is used to improve the detection accuracy, and it can realize real-time online detection of missing bolts on the train bottom, reducing the cost of manual detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of wheel sensor installation.

[0014] Figure 2 Schematic diagram of the installation of the line scan camera module.

[0015] Figure 3 Annotation map for vehicle bottom image dataset.

[0016] Figure 4 Diagram of the target detection algorithm model training process. DETAILED DESCRIPTION

[0017] The present invention provides a method for detecting missing bolts on a train bottom based on image processing, comprising the following steps:

[0018] Step 1: Capturing the original image of the train bottom: Multiple linear array cameras are used to obtain the original image of the train bottom, while two sets of dual-magnetic steel speed measurement methods are used to obtain the train speed;

[0019] Step 2: Distortion correction of the original vehicle bottom image: The original vehicle bottom image is distorted by compression or stretching due to the mismatch between the vehicle speed and the camera scanning rate. A speed curve is fitted based on the acquired vehicle speed data using arc curve fitting technology. The original vehicle bottom image is then subjected to distortion correction by stretching or compressing the fitted curve.

[0020] Step 3: Correcting uneven illumination: Using a two-dimensional Gamma function adaptive correction algorithm, perform uneven illumination correction on the distortion-corrected image to obtain a vehicle bottom image after uneven illumination correction.

[0021] Step 4: Stitching images from multiple camera modules: The FAST and SURF algorithms are combined to extract and describe feature points from the vehicle bottom image after uneven illumination correction. Then, an improved kd-tree nearest neighbor search algorithm (BBF) is combined with an adaptive threshold method for bidirectional matching to stitch the vehicle bottom images collected by different cameras after uneven illumination correction to obtain a complete image of the vehicle bottom.

[0022] Step 5: Missing bolt detection: Create a dataset and train a model based on the acquired complete underbody image set, and use the deep learning target detection algorithm (SSD) to detect missing bolts on the underbody.

[0023] Furthermore, step 1 specifically includes the following steps:

[0024] Two sets of wheel sensors for measuring vehicle speed are installed on the rail side with a mounting fixture. Each set of wheel sensors consists of two, 50 mm from the rail surface, with a 250 mm interval between each pair. The first and second sets of wheel sensors are spaced 13,200 mm apart. The signal lines of the wheel sensors are connected to the PLC, which receives the signals collected by the wheel sensors.

[0025] Multiple linear array cameras with internal integrated compensation light sources for collecting original images of the train bottom are placed in the middle of the track. They are divided into left, middle and right cameras to collect the left, middle and right parts of the train bottom image respectively. The linear array cameras are connected to the host computer for image transmission.

[0026] Furthermore, the specific operations of step 2 are as follows:

[0027] Step 2.1: Use two sets of dual magnets to measure the train speed and obtain 12 sets of discrete speed data.

[0028] Step 2.2: for the 12 sets of discrete speed data, obtain the train speed curve using arc curve fitting technology;

[0029] In step 2.3, the fitted vehicle speed curve is sampled at a sampling interval of 0.1 seconds. The original image of the vehicle bottom is subjected to distortion correction by stretching or compressing according to the fitted curve, as follows:

[0030] Assume that the speed detected by the linear array camera for the first time is v0. Based on this speed v0, the corresponding linear array camera shooting frequency is f0. The width of the original image of the vehicle bottom collected is W and the height is H. The camera resolution used is 4K, the width W is 4096 pixels, and the height H is expressed as follows:

[0031]

[0032] Among them, L is the physical length of the original image of the vehicle bottom, fy is the vertical resolution of the line scan camera.

[0033] Let H i is the height of the i-th image taken by the line array camera, where 0≤i≤N-2, H N-1 is the height of the last image, and the acquisition time is no more than 0.1 seconds, then the following formula is obtained:

[0034]

[0035] After distortion correction, H j Corrected to H tj :

[0036]

[0037] After correction, the height of each image is corrected from H to H T The above correction method plays an important supporting role in subsequent image stitching and bolt detection.

[0038]

[0039] Furthermore, the specific operations of step 3 are as follows:

[0040] Step 31, extracting the brightness component of the distortion-corrected image, specifically as follows:

[0041] F(x,y) represents a two-dimensional function of an image. The value of F(x,y) is the brightness value of the point (x,y) in the image. According to the imaging principle, the image consists of a brightness component and a reflection component of the actual object, as shown in the formula:

[0042] F(x,y)=I(x,y)·R(x,y)

[0043] Among them, I(x,y) is the illumination component and R(x,y) is the reflection component;

[0044] Let P be the image to be corrected for uneven illumination, I be the guide image, and q be the result image. The following transformation relationship exists:

[0045]

[0046] Among them, k represents a pixel point in the image, r represents the radius centered at k, ω k represents a window with k as the center and r as the radius; a k 、b k They are all linear transformation factors and are constant within the window.

[0047] From the above formula, we can see that in any window, there is a gradient relationship as shown above, which means that the edge of P and the edge of I have a good correspondence. In order to minimize the gap between P and q, the window ω k The cost function E(a k ,b k ) is as follows:

[0048]

[0049] Among them, ε is to limit a k Too large a parameter, in order to obtain a k and b k The optimal value of is obtained by using the linear regression method:

[0050]

[0051] Among them, |ω| is the number of pixels in the window, μ k is the mean value in I, σ k is the variance in I, Is the window p k The average value of .

[0052] For different windows, they may contain the same pixel, so a k and b k The value of will also change, in order to obtain q i The value of a in different windows centered at the pixel point k 、b k The average value of is used as the parameter to obtain q i :

[0053]

[0054] Step 32: Correct the extracted brightness component using a two-dimensional Gamma function adaptive correction algorithm, as follows:

[0055] The illumination component is extracted and an improvement is made to the brightness correction function based on the two-dimensional gamma function. The improved two-dimensional gamma function is shown in the following formula. The theoretical value range of m is 0 to 255, so the theoretical value range of the illumination coefficient ε is 0 to 1. The present invention uses ε as 0.5 and m as 128 to correct the image and finds that it has a good adaptive correction effect.

[0056]

[0057] Where O(x,y) is the output image, I(x,y) is the brightness of the image, L(x,y) is the brightness component of a point (x,y) in the image, γ is the gamma correction parameter, ε is the illumination coefficient, and m is the mean brightness of an image.

[0058] Step 33: Fusing the corrected brightness component with the input image to synthesize an output image with uniform illumination.

[0059] Furthermore, the specific operations of step 4 are as follows:

[0060] In step 4.1, the FAST algorithm is used to extract feature points from the image. This method aims to improve extraction speed and quality. The FAST algorithm, due to its relatively simple computation, also improves extraction efficiency. The present invention scans each pixel in the image and then determines whether the grayscale values ​​of the 16 surrounding pixels are greater than that of the pixel. This determination is made using the corner point formula in the FAST algorithm, as shown below.

[0061]

[0062] Where N represents the grayscale value, c is the circumference, I(x) represents the grayscale value of a point in c, P is the center of the circle, and I(P) is the grayscale value of the center point; ε d is the threshold and is set to 9. If N exceeds ε d If , then the pixel point is the corner point, the present invention ε d Set to 9.

[0063] Step 4.2: Use SURF algorithm to describe the feature points: extract a feature point as the center of a circle with a radius of 6σ, where σ is the scale of the selected feature point. Then calculate the sum of the horizontal and vertical Haar wavelet responses of all pixels in the neighborhood of the circle, and superimpose the horizontal and vertical Haar wavelet responses of the image to obtain a vector (m w ,θ w ):

[0064]

[0065] Among them, d x and d y are horizontal and vertical directions respectively; m w When the maximum value is taken, θ w This is the main direction for feature point selection.

[0066] Step 4.3: After the main direction is selected, the feature descriptor is calculated, where the feature descriptor is in a rectangle centered on the selected feature point, with a side length of 20s, where s is the scale value of the feature point.

[0067] The vector V4 describing the feature point is as follows:

[0068] V4=(∑d x ,∑|d x |,∑d y ,∑|d y |)

[0069] In step 4.4, after extracting the feature points, determine the similarity between them. Use the BBF algorithm to calculate the distance between the feature points in the reference image and the test image. Assume T is the threshold for the ratio of the closest feature point to the second closest feature point. Then, perform classification statistics to obtain two sets X1 and X2. Calculate their ratios x1 and x2. Calculate the intersection of the two sets. Assume Y is the ratio of each corresponding matching point in the intersection. Find the minimum value in Y, which is the feature matching threshold.

[0070] In step 4.5, the image is stitched using the fade-in and fade-out weighted fusion method, as shown in the following formula:

[0071]

[0072] Where x1 and x2 are both input vehicle bottom images to be stitched, X is the output image, and γ is the weighting factor, where γ is as follows:

[0073]

[0074] Where ω represents the horizontal coordinate distance of the overlapping parts of the two input vehicle bottom images to be stitched, ω d The horizontal coordinate distance between the pixel points of the overlapping part of the two input vehicle bottom images to be stitched and the overlapping part.

[0075] Furthermore, the specific operations of step 5 are as follows:

[0076] Step 51: manually label each complete vehicle bottom image to create a PASCAL VOC dataset of the complete vehicle bottom image.

[0077] Step 52: Perform model training on the prepared data set and use the deep learning target detection algorithm SSD algorithm to detect the missing bolts on the vehicle bottom. The details are as follows:

[0078] A default prediction bounding box is set during model training. The positive and negative samples are identified based on the intersection-over-union ratio of the default prediction bounding box and the true target bounding box. When the default prediction bounding box contains the target object, it is a positive sample. When the default prediction bounding box does not contain the target object, it is a negative sample.

[0079] The image contains more background parts that are not related to the target than the target, which will cause the sample ratio to be unbalanced during the detection process. In order to keep the sample ratio in a balanced state and make the positive and negative samples 1:3, the SSD algorithm sorts the negative samples and selects a portion of the default predicted bounding boxes that meet the requirements as negative samples. The basis for sorting is the confidence difference between the default predicted bounding box and the true target bounding box.

[0080] For positive samples, multiple default prediction bounding boxes may appear on an object at the same time. In this case, the SSD algorithm uses the non-maximum suppression algorithm to improve it. Default prediction bounding boxes greater than the confidence threshold are considered to contain the target object, while default prediction bounding boxes less than the confidence threshold are considered not to contain the target object.

[0081] During the SSD target detection process, there will be K default prediction bounding boxes on a feature point. Assuming that the target object has a total of C categories, each default prediction bounding box will predict the score of the target object belonging to each analogy and the offset of the four default prediction bounding boxes. Assuming that the feature map size is M×N, the prediction value output by the SSD network is (C+4)×K×M×N; the size of the default prediction bounding box is shown in the formula:

[0082]

[0083] Among them, m represents multiple feature maps, S k is the ratio of the default prediction box in the feature map, S max 、S min Respectively represent S k The maximum and minimum values ​​of the default predicted border width and height are as follows:

[0084]

[0085] in, is the default prediction border width, is the height of the default prediction box, a r is the aspect ratio.

[0086] The present invention uses a linear array camera to capture images of the train's underbody, and then uses image processing technology to obtain a complete image of the subway's underbody. It then identifies faults in key bolts in the underbody image to determine whether there are faults such as bolt detachment. This method is highly timely and has high detection accuracy. Train inspectors can also determine whether there are faults based on the complete underbody image, which is very important for the safe operation of the subway and is safe and reliable.

[0087] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0088] Example

[0089] The technical solution to achieve the purpose of the present invention is: a method for detecting missing bolts on a train bottom based on image processing, comprising the following steps:

[0090] Step 1: Use multiple linear array camera modules to obtain the original image of the train bottom and simultaneously use two sets of dual magnetic steel speed measurement methods to obtain the train speed;

[0091] Two sets of wheel sensors (two in one set) for measuring vehicle speed are installed on the rail side with the installation fixture, 50 mm away from the rail surface. The interval L1 between the two wheel sensors in each set is 250 mm, and the interval L2 between the first and second sets of wheel sensors is 13200 mm. Figure 1 As shown in the figure, S1 is the distance between two relatively close wheels of a subway car, and S2 is the distance between two relatively far wheels of a subway car.

[0092] The linear array camera with internal integrated compensation light source for bottom image acquisition is placed in the middle of the track, and is divided into three cameras: left, middle and right, which respectively acquire the left, middle and right parts of the bottom image of the train. Figure 2 As shown, the line array camera is connected to the host computer for image transmission.

[0093] Step 2: The acquired vehicle bottom image may be compressed or stretched due to the mismatch between the vehicle speed and the camera scanning rate. Based on the 12 sets of vehicle speed data, a speed curve is fitted using arc curve fitting technology. The original vehicle bottom image is stretched or compressed according to the fitted curve to correct the distortion.

[0094] Step 2.1, using two sets of dual magnetic steels to measure the train speed and obtain 12 sets of speed data;

[0095] Step 2.2, using arc curve fitting technology to obtain the train speed curve for the 12 sets of discrete speed data;

[0096] Step 2.3: Sample the fitted speed curve at a sampling interval of 0.1 seconds, and then perform distortion compensation on the image based on the calculation. Assume that the speed detected by the camera for the first time is v0. At this time, the camera shooting frequency corresponding to this speed is f0. The width of the captured image is W and the height is H. The image width is closely related to the camera resolution. The camera resolution used in this invention is 4K, so the width W is 4096px. The height H is expressed as follows:

[0097]

[0098] Where L is the physical length of the image, f y is the vertical resolution of the camera.

[0099] Let Hi is the height of the i-th image taken by the camera, where 0≤i≤N-2, H N-1 is the height of the last image, and the acquisition time is no more than 0.1 seconds, then the formula is as follows.

[0100]

[0101] After distortion correction, H j Corrected to H tj , as shown in the formula.

[0102]

[0103] After correction, the height of each image is corrected from H to H T The above correction method plays an important supporting role in subsequent image stitching and bolt detection.

[0104]

[0105] Step 3: Use a two-dimensional Gamma function adaptive correction algorithm to correct the uneven illumination of the distortion-corrected image to obtain a high-quality vehicle bottom image;

[0106] F(x,y) is a two-dimensional function used to represent an image. Its value is the brightness value of a point in the image. According to the imaging principle, the image can be regarded as composed of the illumination component, that is, the brightness component, and the reflection component of the actual object, as shown in the formula.

[0107] F(x,y)=I(x,y)·R(x,y)

[0108] Where I(x,y) is the illumination component and R(x,y) is the reflection component. Let P be the image to be corrected, I the guide image, and q the result image. The following transformation relationship exists as shown in the formula.

[0109]

[0110] Among them, k represents a pixel point in the image, r represents the radius centered at k, ω k represents a window with k as the center and r as the radius; a k 、b k are linear transformation factors, which are constants within the window. From the above formula, we can see that in any window, there is a gradient relationship as shown above, which means that the edge of P and the edge of I have a good correspondence. In order to minimize the gap between P and q, the window ω k The cost function required is shown in the formula.

[0111]

[0112] Among them, ε is to limit a k Too large a parameter, in order to obtain a k and b k The optimal value of is obtained by using the linear regression method:

[0113]

[0114] Among them, |ω| is the number of pixels in the window, μ k is the mean value in I, σ k is the variance in I, Is the window p k For different windows, they may contain the same pixel, so a k and b k The value of will also change, in order to obtain q i The value of a in different windows centered at the pixel point k 、b k The average value of is used as the parameter to obtain q i :

[0115]

[0116] The illumination component is extracted and improved based on the brightness correction function of the two-dimensional gamma function. The improved two-dimensional gamma function is shown in the following formula. The theoretical value range of m is 0 to 255, so the theoretical value range of the illumination coefficient ε is 0 to 1. In this paper, ε is set to 0.5 and m is set to 128 to correct the image, and it is found that good adaptive correction effect is achieved.

[0117]

[0118] Where O(x,y) is the output image, I(x,y) is the brightness of the image, L(x,y) is the brightness component of a point (x,y) in the image, γ is the gamma correction parameter, ε is the illumination coefficient, and m is the mean brightness of an image.

[0119] Step 4: Combine the FAST algorithm (accelerated segmentation detection feature algorithm) and the SURF algorithm, taking advantage of their respective advantages to extract and describe feature points from the image. Then, combine the improved kd-tree nearest neighbor search algorithm (BBF algorithm) and the adaptive threshold method for bidirectional matching to stitch the calibrated vehicle bottom images captured by different cameras to obtain a high-quality, complete image of the vehicle bottom.

[0120] Step 4.1: Use the FAST algorithm to extract feature points from the image. This aims to improve extraction speed and quality. The FAST algorithm's relatively simple computation also improves extraction efficiency. This paper scans each pixel in the image and then determines whether the grayscale values ​​of the 16 surrounding pixels are greater than that pixel. This determination is made using the corner point formula in the FAST algorithm, as shown in the following formula.

[0121]

[0122] Where N represents the grayscale value, c is the circumference, I(x) represents the grayscale value of a point in c, P is the center of the circle, I(P) is the grayscale value of the center point, ε d is the threshold, if N exceeds ε d If , then the pixel is the corner point, this paper ε d Set to 9.

[0123] Step 4.2: Use SURF algorithm to describe feature points

[45] The algorithm extracts a feature point as the center of a circle with a radius of 6σ, where σ is the scale of the selected feature point, and then calculates the sum of the horizontal and vertical Haar wavelet responses of all pixels in the neighborhood of the circle. The horizontal and vertical directions of the Haar wavelet response of the image are superimposed to obtain the vector (m w ,θ w ).

[0124]

[0125] Among them, d x and d y Respectively horizontal and vertical directions. w When the maximum value is taken, θ w This is the main direction for feature point selection.

[0126] Step 4.3: After the main direction is selected, the feature descriptor is calculated. The feature descriptor is a rectangle centered on the selected feature point, with a side length of 20s, where s is the scale value of the feature point. The vector describing the feature point is shown in the formula.

[0127] V4=(∑d x ,∑|d x |,∑d y ,∑|d y |)

[0128] After extracting the feature points, the similarity between them is determined. The BBF algorithm is used to calculate the distance between the feature points in the reference image and the test image. Assume that T is the threshold for the ratio of the closest feature point to the second closest feature point. Then, classification statistics are performed to obtain two sets X1 and X2. Their ratios are calculated as x1 and x2. The intersection of the two sets is calculated. Assume that Y is the ratio of each corresponding matching point in the intersection. The minimum value in Y is calculated, which is the threshold for feature matching.

[0129] Step 4.4: Use the fade-in and fade-out weighted fusion method to stitch the images, as shown in the following formula.

[0130]

[0131] Where x1 and x2 are both input vehicle bottom images to be stitched, X is the output image, and γ is the weighting factor, where γ is shown in the following formula.

[0132]

[0133] Where ω represents the horizontal coordinate distance of the overlapping parts of the two input vehicle bottom images to be stitched, ω d The horizontal coordinate distance between the pixel points of the overlapping part of the two input vehicle bottom images to be stitched and the overlapping part.

[0134] Step 5: Create a dataset and perform model training on the acquired high-quality complete vehicle bottom image set;

[0135] Step 5.1. Manually label each high-quality complete vehicle bottom image to create your own PASCAL VOC dataset. Use LabelImage Kaiyuan software to label the acquired image set one by one. For the vehicle bottom bolts, there are only two situations: missing and not missing. The label "bolt" means the bolt is not missing, and the label "lose" means the bolt is missing. Figure 3 shown.

[0136] Step 5.2: Perform model training on the prepared data set, such as Figure 4 shown.

[0137] Step 6: Use the deep learning target detection algorithm SSD algorithm to detect the missing bolts on the bottom of the vehicle.

[0138] During network model training, a default predicted bounding box is set. The intersection-over-union (IoU) ratio (IoU) between the default predicted bounding box and the true object bounding box is used to distinguish positive and negative samples. A predicted bounding box containing the target object is considered a positive sample, while a predicted bounding box not containing the target object is considered a negative sample. Generally, images contain more background unrelated to the target than the target itself, which can lead to an imbalance in the sample ratio during detection. To maintain a balanced sample ratio, the ratio of positive to negative samples is generally around 1:3. SSD sorts the negative samples and selects a subset of predicted bounding boxes that meet the requirements as negative samples. The sorting is based on the confidence difference between the predicted bounding box and the true object bounding box. For positive samples, multiple predicted bounding boxes may appear on the same object simultaneously. SSD uses a non-maximum suppression algorithm to improve this situation. The confidence threshold is a critical dividing line in the object detection process. Predicted bounding boxes with a confidence level greater than the threshold are considered to contain the target object, while predicted bounding boxes with a confidence level less than the threshold are considered not to contain the target object.

[0139] During SSD object detection, a feature point is predicted with K bounding boxes. Assuming there are C categories of objects, each predicted bounding box predicts the object's score for each category and the offsets of four comparison predicted bounding boxes. Assuming the feature map size is M×N, the SSD network outputs (C+4)×K×M×N predicted values. The default predicted bounding box size is shown in the formula.

[0140]

[0141] Among them, m represents multiple feature maps, S k is the ratio of the default prediction box in the feature map, and S max 、S min Respectively represent S k The maximum and minimum values ​​of .

[0142] The default prediction box width is as shown in the formula, and the height is as shown in the formula:

[0143]

[0144] is the width of the predicted border, is the height of the predicted border, a r is the aspect ratio.

[0145] The present invention uses a linear array camera with an internally integrated compensation light source for image acquisition. Compared with traditional area array cameras, linear array cameras use linear scanning, so the lateral resolution is high, and the trigger frequency can be changed during the acquisition process, which can also reduce distortion. It also has the advantages of high clarity and high precision. In addition, due to the limitations of on-site hardware equipment installation conditions, the compensation light source does not have good installation conditions. The ET series line scanning module products solve this problem very well. The module integrates a linear array camera and a high-power light source. The image is corrected for distortion using speed curve fitting, which solves the image distortion that occurs during the image acquisition process, and the image is corrected for uneven illumination, which solves the image quality problem caused by uneven light source compensation during the image acquisition process. The use of a target detection algorithm with higher accuracy for small target detection improves detection accuracy; it can realize real-time online detection of missing bolts on the train chassis, reducing the cost of manual inspection.

Claims

1. A method for detecting missing bolts on a train chassis based on image processing, characterized in that: Here are the steps: Step 1: Capturing the original image of the train bottom: Multiple linear array cameras are used to obtain the original image of the train bottom, while two sets of dual-magnetic steel speed measurement methods are used to obtain the train speed; Step 2: Distortion correction of the original vehicle bottom image: The original vehicle bottom image is distorted by compression or stretching due to the mismatch between the vehicle speed and the camera scanning rate. A speed curve is fitted based on the acquired vehicle speed data using arc curve fitting technology. The original vehicle bottom image is then subjected to distortion correction by stretching or compressing the fitted curve. Step 3: Correcting uneven illumination: Using a two-dimensional Gamma function adaptive correction algorithm, perform uneven illumination correction on the distortion-corrected image to obtain a vehicle bottom image after uneven illumination correction. Step 4: Stitching images from multiple camera modules: The FAST and SURF algorithms are combined to extract and describe feature points from the vehicle bottom image after uneven illumination correction. Then, an improved kd-tree nearest neighbor search algorithm (BBF) is combined with an adaptive threshold method for bidirectional matching to stitch the vehicle bottom images collected by different cameras after uneven illumination correction to obtain a complete image of the vehicle bottom. Step 5: Bolt loss detection: The dataset is created and model training is performed on the complete set of vehicle bottom images. The deep learning target detection algorithm (SSD) is used to detect missing bolts on the vehicle bottom. Step 2 specifically includes the following steps: Step 21: Use two sets of dual magnets to measure the train speed and obtain 12 sets of discrete speed data. Step 22, for the 12 sets of discrete speed data, obtain a train speed curve using arc curve fitting technology; Step 23: Sample the fitted vehicle speed curve at a sampling interval of 0.1 seconds, and perform distortion correction on the original image of the vehicle bottom by stretching or compressing it according to the fitted curve, as follows: Assume that the speed detected by the linear array camera for the first time is v0. Based on this speed v0, the corresponding linear array camera shooting frequency is f0. The width of the original image of the vehicle bottom collected is W and the height is H. The camera resolution used is 4K, the width W is 4096 pixels, and the height H is expressed as follows: Among them, L is the physical length of the original image of the vehicle bottom, f y is the vertical resolution of the line array camera; Let H i is the height of the i-th image taken by the line array camera, where 0≤i≤N-2, H N-1 is the height of the last image, and the acquisition time is no more than 0.1 seconds, then the following formula is obtained: After distortion correction, H j Corrected to H tj : After correction, the height of each image is corrected from H to H T : Step 4 specifically includes the following steps: In step 41, each pixel in the vehicle bottom image after the uneven illumination correction is scanned, and then the grayscale values ​​of the 16 pixels around each pixel are determined to be greater than that pixel. The corner point formula of the FAST algorithm is used for determination: Where N' represents the gray value, c is the circumference, I(x) represents the gray value of a point in c, P is the center of the circle, and I(P) is the gray value of the center point; ε d is the threshold and is set to 9. If N' exceeds ε d If , then the pixel is the corner point; Step 42, using SURF algorithm to describe the feature points: extract a feature point as the center of a circle with a radius of 6σ, where σ is the scale of the selected feature point, and then calculate the sum of the horizontal and vertical Haar wavelet responses of all pixels in the neighborhood of the obtained circle, and superimpose the horizontal and vertical Haar wavelet responses of the image to obtain a vector (m w ,θ w ): Among them, d x and d y are horizontal and vertical directions respectively; m w When the maximum value is taken, θ w That is the main direction of feature point selection; Step 43, calculate the feature descriptor, where the feature descriptor is a rectangle centered on the selected feature point, with a side length of 20s, where s is the scale value of the feature point; the vector V4 describing the feature point is shown as follows: V4=(∑d x ,∑|d x |,∑d y ,∑|d y |) Step 44: After extracting the feature points, the similarity between the feature points is determined. The distance between the feature points of the reference image and the detection image is calculated using the BBF algorithm. Assume that T is the threshold of the ratio of the nearest feature point to the second nearest feature point. Then, classification statistics are performed to obtain two sets X1 and X2. The intersection of the two sets is calculated. Assume that Y is the ratio of each corresponding matching point in the intersection. The minimum value in Y is found, which is the threshold for feature matching. Step 45: Use the fade-in fade-out weighted fusion method to stitch the images, as shown in the following formula: Where x1 and x2 are both input vehicle bottom images to be stitched, X is the output image, and γ is the weighting factor, where γ is as follows: Where ω represents the horizontal coordinate distance of the overlapping parts of the two input vehicle bottom images to be stitched, ω d The horizontal coordinate distance between the pixel points of the overlapping part of the two input vehicle bottom images to be stitched and the overlapping part.

2. The train underbody bolt loss detection method based on image processing according to claim 1 is characterized in that: Step 1 specifically includes the following steps: Two sets of wheel sensors for measuring vehicle speed are installed on the rail side with a mounting fixture. Each set of wheel sensors consists of two, 50 mm from the rail surface, with a 250 mm interval between each pair. The first and second sets of wheel sensors are spaced 13,200 mm apart. The signal lines of the wheel sensors are connected to the PLC, which receives the signals collected by the wheel sensors. Multiple linear array cameras with internal integrated compensation light sources for collecting original images of the train bottom are placed in the middle of the track. They are divided into left, middle and right cameras to collect the left, middle and right parts of the train bottom image respectively. The linear array cameras are connected to the host computer for image transmission.

3. The train underbody bolt loss detection method based on image processing according to claim 1 is characterized in that: Step 5 specifically includes the following steps: Step 51, manually annotate each complete vehicle bottom image to create a PASCAL VOC dataset belonging to the complete vehicle bottom image; Step 52: Perform model training on the prepared data set and use the deep learning target detection algorithm SSD algorithm to detect the missing bolts on the vehicle bottom. The details are as follows: A default prediction bounding box is set during model training. The positive and negative samples are identified based on the intersection-over-union ratio of the default prediction bounding box and the true target bounding box. When the default prediction bounding box contains the target object, it is a positive sample. When the default prediction bounding box does not contain the target object, it is a negative sample. The image contains more background parts that are not related to the target than the target, which will cause the sample ratio to be unbalanced during the detection process. In order to keep the sample ratio in a balanced state and make the positive and negative samples 1:3, the SSD algorithm sorts the negative samples and selects a portion of the default predicted bounding boxes that meet the requirements as negative samples. The basis for sorting is the confidence difference between the default predicted bounding box and the true target bounding box. For positive samples, multiple default prediction bounding boxes may appear on an object at the same time. In this case, the SSD algorithm uses the non-maximum suppression algorithm to improve it. Default prediction bounding boxes greater than the confidence threshold are considered to contain the target object, while default prediction bounding boxes less than the confidence threshold are considered not to contain the target object. During the SSD target detection process, there will be K default prediction bounding boxes on a feature point. Assuming that the target object has C categories, each default prediction bounding box will predict the score of the target object belonging to each analogy and the offset of the four default prediction bounding boxes. Assuming the feature map size is Then the predicted value output by the SSD network is The default prediction box size is as shown in the formula: Among them, m represents multiple feature maps, S k is the ratio of the default prediction box in the feature map, S max 、S min Respectively represent S k The maximum and minimum values ​​of the default predicted border width and height are as follows: in, is the default prediction border width, is the height of the default prediction box, a r is the aspect ratio.

Images