Track bolt detection method and system

Abstract

The invention belongs to the field of bolt defect detection, and particularly relates to a track bolt detection method and system. According to the obtained bolt real image of each track bolt to be detected on the production line, the geometric appearance size value of each bolt is obtained through bolt edge information obtained through edge detection; whether the size of each bolt is qualified or not is judged according to whether the geometric appearance size value of each bolt meets the set size qualification condition or not; bolts with unqualified sizes are judged as unqualified products; and respectively inputting the bolt real image of each to-be-detected bolt with the qualified size into the trained defect detection model, and according to a defect detection result output by the defect detection model, obtaining a judgment result containing whether each bolt with the qualified size has a defect or not and a defect position of the bolt with the defect according to the judgment result. The detection process almost does not need manual participation, and the method can guarantee the accuracy and reliability of the detection result on the basis of guaranteeing the high detection efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of bolt defect detection, and in particular relates to a rail bolt detection method and system. Background Art

[0002] With the rapid development of the rail transit industry, the requirements for the safety and reliability of rail transit vehicles are becoming increasingly stringent. Bolts are important fasteners connecting various components of rail vehicles. Their dimensional accuracy and surface quality are directly related to the safe operation of the vehicle.

[0003] For rail bolts, the abnormal problems they often face include surface defects and abnormal geometric dimensions. Surface defects include defects, scratches, wear, and abnormal deformation of the bolt head and threads; abnormal geometric dimensions of the bolts include abnormal dimensions of the bolt head and rod, abnormal verticality of the screw, etc. If the bolts have surface defects, it is very easy to cause the risk of fracture, corrosion failure, loosening due to loss of preload, and affect the smoothness of the track during use, thereby affecting the safety of rail transit; if the bolt size is abnormal, it will affect the close fit between the sleeper and the rail, resulting in insufficient preload, thereby causing the risk of derailment. Therefore, the inspection of rail bolts (including size inspection and surface defect inspection) is the "fuse" for the safe operation of vehicles. The transformation from passive response to active prevention of problems caused by surface defects and size abnormalities of rail bolts is an inevitable choice for the high-quality development of railways.

[0004] In order to prevent the above abnormal problems from affecting rail transit safety in actual applications and causing greater losses, it is necessary to conduct size and defect inspections on the bolts to ensure that the dimensional accuracy and surface quality of the bolts meet the requirements.

[0005] However, the current method for bolt size and defect detection mainly relies on manual sampling or offline testing. This method is not only inefficient, but also greatly affected by human factors. The detection accuracy is unstable and prone to missed detection and false detection, which makes it difficult to meet the actual needs of rail bolt detection. Summary of the Invention

[0006] The object of the present invention is to provide a method and system for detecting rail bolts, which are used to solve the problems in the prior art of low efficiency and unstable detection accuracy of size detection and surface defect detection of bolts on a rail bolt production line.

[0007] To achieve the above-mentioned objectives, the present invention provides a rail bolt inspection method. The method obtains the geometrical appearance dimension value of each bolt based on the real bolt image of each rail bolt to be inspected on the production line and the bolt edge information obtained by edge detection. The method then determines whether the dimension of each bolt is qualified based on whether the geometrical appearance dimension value of each bolt meets the set dimensional qualification conditions. Bolts with unqualified dimensions are judged as unqualified products.

[0008] The real bolt images of each rail bolt to be inspected with qualified size are input into the trained defect detection model respectively. According to the defect detection results output by the defect detection model, a judgment result including whether each bolt with qualified size has defects is obtained, so that the bolts without defects are judged as qualified products.

[0009] Beneficial effects: The present invention provides a new method for detecting rail bolts. The method first performs geometric dimension detection on each bolt to be detected on the production line. The detection method is: whether the geometric appearance dimension value obtained after edge calculation of the real image of each bolt to be detected on the production line meets the set dimension qualification conditions to determine whether the size of the bolt to be detected is qualified. If the bolt size is qualified, the real image of the bolt is input into the trained defect detection model, and the model is finally used to determine whether each bolt that has been determined to be qualified in size has defects, and if it is determined that there are defects, the location of the defects is further determined. In summary, it is equivalent to performing two-level automatic screening on the bolts to be detected on the production line through this method. The first level of screening is achieved through geometric dimension detection. After this detection, bolts with qualified dimensions can be screened out; the second level of screening is achieved through defect detection. After this detection, bolts without surface defects can be screened out from bolts with qualified dimensions. The bolts obtained through these two levels of screening can meet the requirements in terms of dimensional accuracy and surface quality. The entire inspection process involved in this method requires almost no human intervention, which not only ensures high inspection efficiency, but also avoids errors that often occur in manual inspection (i.e., unstable inspection accuracy due to human factors, or problems such as missed inspection and false inspection). In addition, this method can meet the inspection needs of large quantities of bolts to be inspected on the production line. In other words, this method can ensure the accuracy and reliability of the inspection results while ensuring high inspection efficiency. Moreover, if the bolt size is unqualified, such bolts will be directly judged as unqualified products, and no subsequent inspection operations will be performed on them, which can improve efficiency and save inspection costs.

[0010] Further, the training manner of the defect detection model comprises: inputting image samples of track bolts in a training set for training the defect detection model into the defect detection model, first extracting features of the input images, then preliminarily determining ROI regions with defects in the input images according to the extracted features, and obtaining the defect detection result by performing position-sensitive ROI average pooling operation, classification operation and regression operation on the ROI regions.

[0011] Further, the bolt real image comprises: a real image of a bolt head and a real image of a bolt rod.

[0012] Further, the acquisition manner of the training set for training the defect detection model comprises: generating images of the head of the track bolt on the production line with defects according to bolt real images of the head of the track bolt on the production line without defects by using a trained generative adversarial network corresponding to the bolt head; and generating images of the rod of the track bolt on the production line with defects according to bolt real images of the rod of the track bolt on the production line without defects by using a trained generative adversarial network corresponding to the bolt rod.

[0013] The generated images of the head of the track bolt on the production line with defects and the generated images of the rod of the track bolt on the production line with defects are added to the training set for training the defect detection model as image samples of the real image of the bolt head and image samples of the real image of the bolt rod respectively, so as to expand the training set.

[0014] Further, the feature extraction manner of the input image comprises: extracting features of the input image by using a ResNet-101 convolutional neural network.

[0015] Further, the preliminary determination manner of the ROI region with defects in the input image according to the extracted features comprises: extracting the ROI region with defects in the input image by using an RPN network.

[0016] Further, the acquisition manner of the bolt real image of each track bolt to be detected on the production line comprises: performing image enhancement, smoothing denoising processing and processing of separating a bolt body region in the image from a background in which the bolt is located on the original image of each track bolt to be detected on the production line collected by an image collection device, so as to obtain a bolt real image of the track bolt to be detected, remove a region corresponding to the background and retain a region corresponding to the bolt.

[0017] Further, the method further comprises: obtaining a defect type and a defect position of a bolt with a determination result of existing defects according to the defect detection result output by the defect detection model.

[0018] The defect type is used to classify the defects of all defective bolts so as to collect statistics on the types of defects that occur in the defective bolts.

[0019] The defect position is used to locate the defect of each defective bolt respectively, so as to make statistics on the defect positions of the defective bolts.

[0020] The present invention also provides a rail bolt detection system, comprising a processor, wherein the processor is configured to execute a computer program to implement the steps of the above-mentioned rail bolt detection method.

[0021] The rail bolt detection system can achieve the same beneficial effects as the above-mentioned rail bolt detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a method for detecting rail bolts in an embodiment of the method for detecting rail bolts of the present invention;

[0023] Figure 2 A flowchart of Canny edge detection in an embodiment of a method for detecting rail bolts of the present invention;

[0024] Figure 3 Schematic diagram of the training process of the defect detection model in the embodiment of the rail bolt detection method of the present invention;

[0025] Figure 4 Schematic diagram of the process of generating an image of a rail bolt (head and shank) with defects on a production line using a Pix2pix network model in an embodiment of the rail bolt detection method of the present invention;

[0026] Figure 5 This is a classification diagram of the image data set corresponding to the rail bolts in the implementation method of the rail bolt detection method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and implementation methods.

[0028] Implementation method for detecting rail bolts

[0029] This embodiment provides a technical solution for a method for detecting rail bolts. The main concept of this solution is: to perform two types of inspections on the rail bolts to be inspected on the production line: automated geometric dimension inspection and automated (surface) defect inspection. There is a sequence for these two inspections, that is, geometric dimension inspection is performed first, and bolts that fail the dimension inspection are screened out and do not participate in subsequent defect inspection; then, defect inspection is performed on bolts with qualified dimensions using a trained defect detection model, and finally it is determined whether each bolt that meets the dimensional accuracy requirements has surface defects, so as to achieve the effect of ensuring efficient inspection of rail bolts on the production line and accurate and reliable inspection results of the bolts, and finally screening out bolts with qualified dimensions and no defects (that is, quality that meets the requirements).

[0030] refer to Figure 1 In this embodiment, the method includes: obtaining a real image of each rail bolt to be inspected on the production line (the real image of the bolt includes a real image of the bolt head and a real image of the bolt shank), and obtaining the geometric appearance dimension value of each bolt through the bolt edge information obtained by edge detection; judging whether the dimension of each bolt is qualified based on whether the geometric appearance dimension value of each bolt meets the set dimension qualification condition; and judging bolts with unqualified dimensions as unqualified products.

[0031] In one embodiment of the present embodiment, the method for acquiring the real image of the bolt of each rail bolt to be inspected on the production line is as follows: the conveyor belt and the manipulator cooperate to capture images of various parts of the bolt, and the real image of the bolt product after processing on the production line is acquired in real time in this way. After obtaining the real image of the bolt, the camera (i.e., the image acquisition device corresponding to the real image of the bolt) is calibrated to obtain the conversion relationship between the pixel coordinate system and the world coordinate system and the bolt edge information (obtained by edge detection), and the pixel size of each feature in the image is converted into the size in the real world, so as to obtain the geometric appearance size value of the actual bolt. After obtaining the geometric appearance size value, the real image of the bolt head and the real image of the bolt shank are respectively subjected to geometric appearance size detection, and the content of the detection includes:

[0032] According to whether each bolt respectively meets a set size qualification condition (which can be set according to different specifications and requirements of each production line), it is determined whether each bolt is qualified. For example, a production line produces a batch of metric hexagonal head bolts, and the set size qualification condition includes that the diameter d of the bolt shank satisfies 7.8≤d≤8.2 (unit: mm), and the width D of the opposite sides of the bolt head (which is usually 1.5 times the diameter of the bolt shank) satisfies 11.75≤D≤12.25 (unit: mm). If the diameter d of the bolt to be measured is 8.05 mm and the width D of the opposite sides of the bolt head is 11.98 mm, the bolt meets the above size qualification condition, and the bolt is qualified. If the diameter d of the bolt to be measured is 8.23 mm and the width D of the opposite sides of the bolt head is 12.3 mm, the diameter d of the bolt shank does not meet the set size qualification condition, and the bolt is unqualified. Such unqualified bolts are determined as unqualified products and are not subjected to subsequent detection. This operation is equivalent to eliminating the bolts that do not pass the geometric appearance size detection, and the eliminated bolts are not used as test samples for defect detection. Through such a processing manner, the number of test samples can be reduced, and the detection efficiency can be improved to avoid meaningless detection.

[0033] In the present embodiment, after obtaining the detection result of whether the bolt size is qualified, the bolt real image of each size-qualified bolt to be detected is input into the trained defect detection model. The defect detection result output by the defect detection model is obtained to determine whether each size-qualified bolt has a defect, so as to determine the bolt without a defect as a qualified product. It should be noted that the geometric size detection and the surface defect detection in the present embodiment have a set order, i.e., the geometric size detection is performed first, and the actual input of the defect detection model is determined according to the output result of the geometric size detection.

[0034] In a preferred embodiment of the present embodiment, the geometric appearance size value of the actual bolt is obtained by cooperation of the conveying belt and the mechanical arm, photographing the bolt head and the shank (i.e., the real image of the bolt head and the real image of the bolt shank), and then performing Canny edge detection on the preprocessed image. In other embodiments, other existing edge detection algorithms can also be used.

[0035] In the present embodiment, the bolt real image of each bolt to be detected on the production line is obtained by cooperation of the conveying belt and the mechanical arm, photographing the original image of the bolt head and the original image of the bolt shank, and then preprocessing the images to obtain the preprocessed image as the bolt real image.

[0036] The pre-processing (i.e. initialization operation) includes: performing gray scale conversion, image enhancement, smoothing and denoising processing on the original image of each rail bolt to be detected on the production line collected by the image collection device, and processing to separate the bolt body region in the image from the background where the bolt is located, to obtain a bolt real image of the bolt to be detected, to remove the region corresponding to the background and retain the region corresponding to the bolt.

[0037] In one preferred embodiment of the present embodiment, the specific manner of image enhancement is to enhance the image effect by using histogram equalization and contrast enhancement. In order to reduce the difficulty of geometric and defect detection, the image is first converted into a gray scale image using a weighted average method, and the conversion formula is as follows:

[0038] H = 0.2989 * R + 0.587 * G + 0.114 * B

[0039] In the above formula, H represents the gray value; R represents the red channel value (0-255) of the pixel in the input image; G represents the green channel value (0-255) of the pixel in the input image; and B represents the blue channel value (0-255) of the pixel in the input image.

[0040] After the gray scale conversion is completed, the contrast is enhanced by histogram equalization;

[0041] That is, the pixel distribution of the bolt image is known, and the statistical histogram h(k) defines the number n of pixels of each gray level k :

[0042] h(k) = n k (k = 0, 1, 2,..., 255)

[0043] The cumulative distribution function f(k) of the histogram is then calculated:

[0044]

[0045] The cumulative distribution function f(k) is used to map each gray level s to a new gray level r:

[0046]

[0047] The original gray level s in the bolt image is converted into a gray level r and replaces the original pixel value. The equalized image histogram h'(r) defines the number n of pixels of the new gray level r r :

[0048] h'(r) = n r (r = 0, 1, 2,..., 255)

[0049] Histogram equalization can adjust the grayscale distribution of the image to make the grayscale value distribution of the pixels more uniform, thereby enhancing the brightness of the bolt image and enhancing the edges and details of the bolt image to make them more obvious.

[0050] After using the above method to enhance the image, a median filter is used to smooth the image and remove noise to improve image quality. The specific method involves selecting a 3x3 convolution kernel, applying the kernel to every pixel in the bolt image, sorting the pixels within the window in ascending or descending grayscale order, and replacing the grayscale value of that pixel with the median value. This process is repeated for all pixels in the image to remove image noise and complete the image processing.

[0051] After smoothing and denoising, the maximum between-class variance method (OTSU) is used for threshold segmentation to separate the bolts from the background, retaining the region of interest and reducing the difficulty of subsequent bolt detection. The specific methods include:

[0052] Using the maximum inter-class variance method for threshold segmentation can distinguish the entire bolt image from the background. The optimal threshold of the image is set to t. According to the statistical histogram h'(r), the total number of pixels is m×n, and the probability of each gray level is:

[0053]

[0054] Through this operation, the image is divided into two parts: bolt target and background, where the number of bolt target pixels accounts for p1 in the image and the average gray level is μ1.

[0055] The ratio of background pixels to image is p2, the average gray level is μ2, and the inter-class variance is σ 2 Initialize the target and background image proportions and mean grayscale, initialize the proportion, mean and variance to 0, and then through continuous iteration, find the solution that makes σ 2 Get the maximum t value, that is:

[0056] p1=p1+h'(t)

[0057] p2=m*n-p1

[0058] μ1=μ1+t*p(t)

[0059] μ2=μ2+t*p(t)

[0060] σ 2 (t) = p1p2(μ1-μ2) 2

[0061] Find such that σ 2 (t) Maximum t * , that is, the optimal threshold t* , and then perform the following binarization processing:

[0062]

[0063] In the above formula, f'(i, j) is the pixel value at position (i, j) in the original image, and g(i, j) is the pixel value after threshold segmentation. Connected domains are determined for the binarized region, redundant regions are removed, and the bolt target image is segmented from the background to extract the bolt target image.

[0064] In other embodiments, other existing image processing methods may also be used, which will not be described in detail here.

[0065] Specifically, after obtaining the real image of the bolt head and the real image of the bolt shank, in order to determine the geometric appearance size of the bolt in the image, it is necessary to calibrate the camera first (the corresponding camera calibration result will be obtained after calibration). After successful calibration, there is no need to calibrate again, and the conversion relationship between the pixel coordinate system and the world coordinate system is established. Let the world coordinate system be O W -X W Y W Z W , the camera coordinate system is O C -X C Y C Z C , the pixel coordinate system is UOV, and the image coordinate system is X P Y P Z P , the coordinates of the target point P(X C ,Y C ,Z C ), the coordinates of the target image point are P i (X i ,Y i ) to determine the ratio of image pixels to the actual world size. The conversion relationship between the pixel coordinate system and the world coordinate system is obtained through coordinate transformation:

[0066]

[0067] Among them, (u, v) is the coordinate of the pixel coordinate system, there are 6 internal parameters, the coordinate axis tilt coefficient is usually 0, and (u0, v0) is the coordinate of the principal point of the image center in the pixel coordinate system. x , d y For each pixel in the image coordinate system X P , Y PThe length in the direction of the image, and f is the focal length. There are six extrinsic parameters: R is a 3x3 matrix for attitude transformation, including parameters for the three-axis rotation angles, and T is a 3x1 matrix for translation transformation, including parameters for the three-axis translation. Using the Zhang Zhengyou calibration method, a calibration plate with known black and white checkerboard dimensions is photographed from different angles and distances with the camera, ensuring that the plate is completely visible in the image. The position of the calibration plate in the camera's field of view is continuously changed, and 10-20 photos are taken to obtain multiple sets of data to determine the extrinsic and intrinsic parameter values. These extrinsic and intrinsic parameter values ​​can be used to calculate the camera's pixel equivalent, determining the actual physical size represented by each pixel in the pixel coordinate system.

[0068] The specific operation used in the above edge detection is: Non-Maximum Suppression (NMS), which is mainly used to refine edges, eliminate redundant responses, and improve the accuracy and clarity of edge detection.

[0069] Because edge detection algorithms (such as the Canny algorithm used in this embodiment) typically produce a wide response area near the edge when detecting an edge, non-maximum suppression compares the gradient magnitudes of adjacent pixels and retains only the local maximum along the gradient direction, thereby refining the edge to a single pixel width.

[0070] The non-maximum suppression in this embodiment includes the following three specific operations:

[0071] ①Compare pixel values ​​along the gradient direction;

[0072] For each pixel, check the values ​​of the two adjacent pixels along its gradient direction (i.e., the direction of the most dramatic brightness change in the image). For example, if the gradient direction is 45° as shown in the figure, compare the gradient values ​​(i.e., gradient magnitudes) of the current pixel (x, y) with the adjacent pixels (x', y') and (x", y") along the gradient direction.

[0073] Before comparing pixel values ​​along the gradient direction, it is necessary to first calculate the gradient (i.e., the rate of grayscale change); the larger the gradient magnitude, the more likely the point is an edge. The calculation method is: use the Sobel operator to obtain the gradient magnitude (i.e., gradient magnitude) and direction of each pixel. By calculating the gradient magnitude and direction of each pixel in the image, the strength and direction of the edge can be characterized, and the derivative of each pixel in the x-direction and y-direction can be obtained:

[0074]

[0075] Among them, G x represents the derivative of each pixel in the x direction; G y Represents the derivative of each pixel in the y direction.

[0076] The gradient magnitude is calculated as follows:

[0077]

[0078] Among them, ||G|| represents the gradient amplitude of each pixel (that is, the size of the gradient); the meanings of other parameters are consistent with the above.

[0079] The gradient direction is calculated as follows:

[0080]

[0081] Among them, θ is the gradient direction angle, which can indicate the direction in which the image brightness changes fastest; the meanings of other parameters are consistent with the above.

[0082] ② Determine whether it is a local maximum;

[0083] If the gradient value of the current pixel is not a local maximum (that is, not greater than the values ​​of the two adjacent pixels), the pixel value is suppressed to zero, which means that it is not considered to be part of the edge. The suppression formula for the image gradient is:

[0084]

[0085] As shown in the above formula, a pixel is retained only if its gradient value is greater than that of its two adjacent pixels in the gradient direction, otherwise its value is set to 0.

[0086] ③ Eliminate redundant responses;

[0087] This step is used to retain precise edges. Through this process, redundant gradient responses that have no actual edge significance can be removed, and only true edge pixels are retained. The edge line is thus refined and the accuracy is greatly improved.

[0088] In this embodiment, after non-maximum suppression is completed, dual-threshold detection and hysteresis connection are performed based on the Canny algorithm. That is, dual thresholds are used to classify edges, and strong edges and weak edges are connected through hysteresis processing. To further remove false edge noise and incomplete edge lines, the following dual-threshold detection and hysteresis connection are used:

[0089] A) Dual threshold detection;

[0090] In the traditional Canny algorithm, two thresholds (a high threshold and a low threshold) are pre-set based on the gradient value of the pixel. Edges are then classified as "strong edges" or "weak edges" based on the relationship between the gradient magnitude and the threshold. Edges above the high threshold are considered strong edges, those below the low threshold are considered non-edges, and those between the two are considered weak edges. Here, by observing the grayscale histogram used in image preprocessing, which displays the number of pixels at each grayscale level in the image, the two main peaks in the histogram are identified. These peaks typically represent the bolt image and the background in the image. The lowest point between the two peaks is used as the high threshold, and a value lower than the high threshold is selected as the low threshold. This method improves the dual-threshold selection in the Canny operator.

[0091] B) Lagged connection and connectivity analysis;

[0092] After detecting the three types of edges through the double threshold method, we further determine which weak edges are actual edges and which are noise through hysteresis connection and connectivity analysis; this involves retaining strong edges, determining weak edges, and discarding non-edges.

[0093] Preservation of strong edges: Pixels with gradient values ​​greater than a high threshold are directly marked as edges. As shown in the figure above, these are the most obvious edge points in the image.

[0094] Weak edge determination: For pixels with gradient values ​​between the high and low thresholds (weak edges), further check whether they are connected to strong edges. If there is a strong edge pixel in the 8-neighborhood of the weak edge pixel (i.e., the surrounding eight pixels), the weak edge pixel is retained as an actual edge.

[0095] Non-edge discarding: Pixels with gradient values ​​below a low threshold are directly discarded and are not considered as edges.

[0096] Iterative processing: This process is performed iteratively, gradually connecting weak edges with strong edges until the edges in the image no longer change. This delayed connection strategy ensures that weak edges are not mistakenly discarded, thus maintaining the integrity of the edges.

[0097] Through dual threshold detection and hysteresis connection, the Canny algorithm can effectively distinguish actual edges from noise, ensuring that important edge information in the image is retained while eliminating unnecessary redundancy.

[0098] The dual thresholding and hysteresis processing described above not only eliminates the effects of noise but also effectively preserves important edge information in the image. Each edge line is continuous and refined, and noise is removed to the greatest extent possible, resulting in a precise binary edge map, which is the final output image of the bolt edge preserved according to the Canny operator.

[0099] In this embodiment, based on the length, width, diameter, and inclination of the bolt contour edge displayed in the bolt edge image retained by the Canny operator, combined with the camera calibration results, the pixel size of each feature in the image is converted into the size in the real world, thereby obtaining the geometric appearance size value of each bolt. After obtaining the geometric appearance size value, the bolt is subjected to geometric dimension detection. The method of geometric dimension detection has been explained above and will not be repeated here. The geometric dimension detection will obtain a detection result, that is, a detection result that determines whether the size of each bolt is qualified; based on the detection result, the real bolt image of each rail bolt to be inspected with qualified size is input into the trained defect detection model to perform (surface) defect detection. In other embodiments of this embodiment, the bolt edge image can also be obtained according to other edge detection methods, and the geometric appearance size value is then obtained based on the bolt edge image.

[0100] In this embodiment, when performing defect detection on the rail bolts to be inspected with qualified dimensions, it involves the construction and training of a defect detection model. In one embodiment of this embodiment, the defect detection model is selected when it is constructed. Figure 3 The R-FCN deep convolutional neural network (i.e., region-based fully convolutional neural network) model shown in FIG. The constructed model is trained to obtain a trained defect detection model.

[0101] When training the constructed defect detection model, the training method includes: using the training set for training the defect detection model (the images contained in the training set can be referred to Figure 3 Image samples of rail bolts (corresponding to the "80% of the training set" in the training data) are fed into the defect detection model. The model first extracts features from the input images and then, based on these features, preliminarily identifies defective ROI regions within the input images. Position-sensitive ROI average pooling, classification, and regression are then performed on these ROI regions to obtain defect detection results. The model is then trained based on the obtained defect detection results and the true labels corresponding to the input samples. The feature extraction method for the input images includes using a ResNet-101 convolutional neural network. The convolutional layers of this convolutional neural network are shown in Table 1.

[0102] Table 1

[0103]

[0104] The method of preliminarily determining the ROI region with defects in the input image based on the features extracted by ResNet-101 includes: extracting the ROI region with defects in the input image through the RPN network.

[0105] The small size of the bolt defect sample dataset affects the accuracy and efficiency of detection model training. Therefore, the dataset needs to be expanded. This expansion method, employed in this implementation, uses a generative adversarial network to generate virtual images as augmented data, adding them to the bolt defect sample dataset to increase the sample size.

[0106] In this embodiment, the training set for training the defect detection model is obtained by: generating images of rail bolt heads on the production line with defects, using a trained generative adversarial network corresponding to the bolt heads, based on real images of rail bolt heads on the production line without defects; and generating images of rail bolt shanks on the production line with defects, using a trained generative adversarial network corresponding to the bolt shanks, based on real images of rail bolt shanks on the production line without defects. The generated images of rail bolt heads on the production line with defects and images of rail bolt shanks on the production line with defects are added to the training set for training the defect detection model as image samples of real images of the bolt heads and real images of the bolt shanks, respectively, to expand the training set.

[0107] In fact, for the generative adversarial network used to generate expanded samples of bolt heads (i.e., the generative adversarial network corresponding to the bolt head) and the generative adversarial network used to generate expanded samples of bolt shanks (i.e., the generative adversarial network corresponding to the bolt shank), the initial network structures of the two are the same, but the two network structures are trained separately. By inputting real defect-free images of bolt heads and real defect-free images of bolt shanks respectively, the generator outputs the generated defect images, and then the discriminator distinguishes the generated defect images from the real defect images, and outputs the probabilities representing the real images and the generated images, and backpropagates the errors to optimize the generator, thereby obtaining different generative adversarial network models.

[0108] In one embodiment of this embodiment, the generative adversarial network used is a generative adversarial network pix2pix model (i.e. Figure 4 The Pix2pix GAN model (shown in Figure 2) consists of a generator and a discriminator. The generator's purpose is to convert images of real, defect-free bolts (i.e., images of rail bolts on a production line with no defects on the head or shank, respectively) into target bolt images with defects. The discriminator receives two inputs: a real, defect-free image and a defective image output by the generator; and a real, defective image and a real, defect-free image. The discriminator's task is to distinguish between the generated and real images, outputting probabilities for the real and generated images, and backpropagating the error to optimize the generator.

[0109] The loss function of the Pix2pix GAN model is defined as:

[0110] Fighting Loss:

[0111] L cGAN (G,D)=E x,y [logD(x,y)+E x,z [log(1-D(x,G(x,z)))]

[0112] L1 pixel-level loss:

[0113] L L1 (G)=E x,y,z [||yG(x,z)||1]

[0114] Total loss:

[0115]

[0116] In the above formula, E represents the expectation, x represents the input image of a true non-defective bolt, y represents the input image of a true defective bolt, and z represents the random noise vector. G(x,z) represents the defective bolt image generated by the generator G based on the input image x and the random noise z. D(x,z) represents the probability of the discriminator D distinguishing between the true image (x,y). D(x,G(x,z)) represents the probability of the discriminator D distinguishing between the generated image G(x,z) and the input image x.

[0117] a) Generator G;

[0118] The U-Net architecture is adopted, which consists of an encoder and a decoder. The encoder uses convolutional layers and downsampling layers to extract the features of the real bolt image, and the decoder uses deconvolution layers and upsampling layers to use the features to generate the target bolt image with defects.

[0119] The loss function of the generator is:

[0120] L G =-E x,z [logD(x,G(x,z))]-λE x,y,z [||yG(x,z)||1]

[0121] In the above formula, the first term is the adversarial loss, which means that the generator G hopes that the generated image can deceive the discriminator D; the second term is the L1 (pixel level) loss, which means that the generator G hopes that the generated image is as close as possible to the real image at the pixel level.

[0122] In the encoder, the convolution layer is used to extract image features, using ReLU as the activation function. Batch normalization is used to accelerate the training process and reduce internal covariate shift. Finally, the size of the feature map is reduced through convolutional and pooling layers. In the decoder, the deconvolution layer is used to gradually restore the size of the image. It also uses ReLU as the activation function and is directly connected from the encoder part to the decoder part through skip connections to retain detailed information and finally output the generated target image.

[0123] b) Discriminator D;

[0124] The Patch-GAN architecture takes in real and generated images, extracts features through a series of convolutional layers, and ultimately outputs a matrix where each element corresponds to the authenticity score of a small patch in the input image. The authenticity score of the entire image is then averaged to obtain the authenticity score. This allows the algorithm to distinguish between real bolt images and generated bolt images.

[0125] The loss function of the discriminator is:

[0126] L D =E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))]

[0127] In the above formula, the first term represents the discriminant loss of the discriminator D for the real image pair (x, y); the second term represents the discriminant loss of the discriminator D for the generated image G(x, z) and the input image x. The goal of the discriminator D is to maximize the sum of these two losses, that is, to improve the ability to distinguish between real images and generated images.

[0128] The training process of the Pix2pix GAN model is as follows: initialize the weights of the generator and discriminator, define the adversarial loss of the generator, L1 loss (pixel-level loss), and the loss of the discriminator.

[0129] The training of the discriminator includes: using real images and generated images as input, the discriminator judges the real images and the generated defect images respectively, and updates the discriminator weights using the Adam optimizer by calculating the loss of the discriminator.

[0130] The training of the generator involves generating defect images based on real bolt images, using the generated images and feedback from the discriminator, and then calculating the generator's loss and updating the generator's weights using the Adam optimizer. This process is repeated until the discriminator cannot distinguish between the generated images and real images, thus completing the expansion of the bolt defect image dataset.

[0131] In this embodiment, after expansion in the above manner, the following is obtained: Figure 5 The expanded bolt image dataset shown in the figure contains more sample data, including: a bolt head image dataset and a bolt shank image dataset; the bolt head image dataset includes 80% of the training set for training the defect detection model (including real images of bolt heads with defects and images of bolt heads with defects generated by a generative adversarial network) and 20% of the test set for testing the trained defect detection model (including real images of bolt heads with defects and real images of bolt heads without defects). The same applies to the bolt shank image dataset. In other embodiments of this embodiment, sample supplementation can also be achieved through other methods (for example, using other generative adversarial networks to generate such bolt images or manually creating such bolt images).

[0132] In one embodiment of this embodiment, after obtaining the above training set and test set, Figure 2 The constructed defect detection model is trained in the following ways:

[0133] 1) Use the ResNet-101 convolutional neural network as the base network to extract the features of the input image;

[0134] In this embodiment, this step is used to extract the features of the input image (i.e., the image included in the training set). The specific operation is as follows: the default image input size of the ResNet-101 convolutional neural network is 224×224, three channels, and the base layer (Conv1_x) uses a 7×7 convolution kernel, the number of convolution kernels is 64, and the step size is 2. The input image is convolved, and then a 3×3 maximum pooling layer is used with a step size of 2 to obtain a 56×56, 64-channel output, further reducing the size of the feature map. The subsequent convolutional layers are composed of three convolutional layers: 1×1, 3×3, and 1×1. The number of residual block bottlenecks contained in each residual convolution layer is 3, 4, 23, and 3 respectively. After passing through the base layer, the image enters the residual layer. Within the residual layer, it undergoes feature processing using residual blocks. It first passes through a 1×1 convolutional layer to reduce the number of channels, then a 3×3 convolutional layer to extract features, and finally another 1×1 convolutional layer to increase the number of channels. Each convolutional layer is followed by batch normalization (BN) and a ReLU activation function (except after the last convolutional layer in Conv5_x). After passing through all the residual layers, the feature map's dimensionality gradually decreases while the number of channels increases, thereby extracting more abstract and rich features.

[0135] 2) Use the RPN network to extract candidate regions of suspected defects on the feature map;

[0136] In this embodiment, this step is used to preliminarily determine the ROI area where defects exist in the input image (i.e., the candidate area of ​​suspected defects). The specific operation is: apply a convolution kernel size of 3×3 on the feature map finally obtained above, and slide a sliding window with a step size of 1 on the feature map, and generate a new feature vector at each position, totaling K fixed-ratio anchor boxes (bounding boxes). For each sliding window position, RPN will perform category judgment and bounding box regression to output the calculated score and the adjusted bounding box position. Category judgment uses a 2K dimensional convolution layer to predict the probability of defects and background for each anchor box to determine whether the position includes defects; bounding box regression uses a 4K dimensional convolution layer to predict the offset of the bounding box for each anchor box (x, y, w, h), determine if the position contains defects, the position and size of the corresponding object bounding box, and obtain the corresponding ROI area.

[0137] 3) Map the candidate region ROI output by the RPN network to the position-sensitive score map, and obtain a fixed-size ROI feature map through the pooling layer;

[0138] In this embodiment, this step is used to perform position-sensitive ROI average pooling, classification, and regression on the ROI region to obtain defect detection results. Specifically, a position-sensitive score map generation layer is added to the feature map obtained by the last convolutional layer to generate k*k grids for each defect partition, with each grid responsible for detecting a specific portion of the corresponding target area. Assuming there are C types of defects, plus background, for a total of C+1 categories, a k*k*(C+1)-dimensional position-sensitive score map is obtained for classification, and a 4*k*k-dimensional position-sensitive score map is obtained for regression.

[0139] This step involves the position-sensitive ROI average pooling operation, which is as follows: the ROI area extracted by the above RPN contains four values ​​of x, y, w, and h. For each different ROI, it is mapped to the position-sensitive score map. An ROI will be divided into k*k sub-regions (bins). The length and width of each sub-region are h / k and w / k respectively. Each bin corresponds to a certain area on the position-sensitive score map. The average pooling operation is performed on the position-sensitive score map sub-region corresponding to each bin to extract a fixed-size feature map. Each ROI will generate a feature map. The position-sensitive ROI pooling operation for the (i, j)th bin is as follows:

[0140]

[0141] In the above formula, r c (i, j) is the pooling result of the (i, j)th bin of the cth category, z i,j,cis one of the k*k*(C+1) score maps, (x0,y0) represents the upper left corner of a ROI, n is the number of pixels in the bin, and θ represents all learnable parameters in the network.

[0142] In this embodiment, the above classification operation is specifically as follows: the number of defect types in the k*k*(C+1)-dimensional position-sensitive score map is (C+1). According to the above pooling method, for each defect type, the ROI can obtain k*k values. Now there are a total of (C+1) defect types, so one ROI can obtain k*k*(C+1) values. The k*k of each type represents the response value of the ROI belonging to that category. Then, adding these k*k numbers will get the score of the category, so there are a total of (C+1) scores. Then, each score is passed through the softmax function to obtain the probability of each defect type, which is used to calculate the cross entropy loss during network training, and then the ROI is ranked by score. The softmax value of each category is calculated as follows:

[0143]

[0144] In the above formula, S c (θ) represents the probability that the current ROI region belongs to category c when the model parameter is θ, c∈{0,1,2,...,C}, where 0 represents background and 1 to C represent defect categories; e rc(θ) represents the index of the pooling score of category c when the model parameter is θ; e rc'(θ) It represents the exponential summation of the pooled scores of all categories (including background) to normalize the probability.

[0145] The above regression operation is specifically as follows: After the 4*k*k-dimensional position-sensitive score map undergoes the same position-sensitive ROI average pooling operation, the final output is that each ROI can obtain 4 values ​​as the x, y, w, and h offsets of the ROI, and finally obtain the position coordinates of the defect (that is, determine the location of the defect).

[0146] Based on the determination result of whether each bolt having a qualified size has a defect, this embodiment further includes: obtaining the defect type and defect location of the bolt having a defect determined to have a defect according to the defect detection result output by the defect detection model;

[0147] Defect type is used to classify the defects of all defective bolts so as to make statistics on the types of defects that occur in defective bolts;

[0148] The defect location is used to locate the defect of each defective bolt, so as to provide statistics on the locations of the defective bolts. In a preferred embodiment of this embodiment, the defect detection model is trained according to the above method. After obtaining the trained defect detection model, the real image of the rail bolt to be inspected is used as the input of the model. The specific detection process of the model is based on the above model training process. The final output of the model is the defect detection result, which includes the following content:

[0149] 1) Determination of whether each bolt with qualified dimensions has defects;

[0150] The judgment result includes two situations: qualified and unqualified. That is, if there are defects, it is judged as unqualified, and if there are no defects, it is judged as qualified.

[0151] II) the defect type and defect location of the bolts determined to be defective;

[0152] The result of the determination is the defect type of the defective bolts (i.e., bolts that have been determined to be unqualified), that is, in this embodiment, the defect types of the bolts can be counted so that technicians can know and count which specific types of bolt defects are included, so as to avoid such defects during production. For example, statistics show that among the defective bolts on a certain production line, 30% have quenching cracks on the head and 40% have dents on the head. This means that there may be problems in a certain process or a certain equipment during production, resulting in a high probability of these two types of defects in the same batch of bolts, accounting for 70% of all defective bolts. Based on this conclusion, technicians can pay more attention to the conditions for the occurrence of these two types of defects, and do preventive work for subsequent production by checking relevant production equipment or further analyzing the causes of defects.

[0153] The determination result is the defect location of the defective bolt. In other words, in this embodiment, the defect location of the bolt can be located, allowing technicians to identify and count the specific defect locations of the bolts, thereby preventing such defects during production. For example, if statistical analysis reveals that 80% of defective bolts on a certain production line have roughly the same defect location, this indicates that a problem occurred in a certain process or equipment during production, resulting in widespread defects in the same batch of bolts, which can be corrected in subsequent production.

[0154] Implementation method of rail bolt detection system

[0155] This embodiment provides a technical solution for a rail bolt detection system. The system includes a processor having executable program instructions stored therein. The executable program instructions are used to implement the rail bolt detection method in the above-mentioned rail bolt detection method embodiment.

[0156] Since the specific working mode and working principle of the rail bolt detection system of this embodiment have been described in detail in the above-mentioned embodiment of the rail bolt detection method, they will not be repeated here.

[0157] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention.

Claims

1. A method for detecting rail bolts, characterized in that: include: Based on the real images of each rail bolt to be inspected on the production line, the geometric dimensions of each bolt are obtained using the bolt edge information obtained through edge detection. The dimensions of each bolt are then determined to be qualified based on whether the geometric dimensions meet the set dimensional qualification criteria. Bolts that do not meet the dimensional qualification criteria are judged as unqualified products. The real bolt images of each rail bolt to be inspected with qualified size are input into the trained defect detection model respectively. According to the defect detection results output by the defect detection model, a judgment result including whether each bolt with qualified size has defects is obtained, so that the bolts without defects are judged as qualified products.

2. The method for detecting rail bolts according to claim 1, characterized in that: The training method of the defect detection model includes: inputting image samples of rail bolts in a training set used to train the defect detection model into the defect detection model, first extracting features of the input image, then preliminarily determining an ROI region where defects exist in the input image based on the extracted features, and obtaining the defect detection result by performing position-sensitive ROI average pooling operations, classification operations, and regression operations on the ROI region; and training the model based on the obtained defect detection results and the true labels corresponding to the input samples.

3. The method for detecting rail bolts according to claim 1 or 2, characterized in that: The real image of the bolt includes: a real image of the bolt head and a real image of the bolt shank.

4. The method for detecting rail bolts according to claim 3, characterized in that: The training set for training the defect detection model is obtained by: using real images of rail bolt heads on the production line without defects, and using a generative adversarial network trained for bolt heads to generate images of rail bolt heads on the production line with defects; using real images of rail bolt shanks on the production line without defects, and using a generative adversarial network trained for bolt shanks to generate images of rail bolt shanks on the production line with defects; The generated images of the rail bolts with defects on the head and the images of the rail bolts with defects on the production line are added to the training set used to train the defect detection model, as image samples of the real images of the bolt heads and the real images of the bolt shafts, respectively, to expand the training set.

5. The method for detecting rail bolts according to claim 2, characterized in that: Methods for extracting features of the input image include: extracting features of the input image through a ResNet-101 convolutional neural network.

6. The method for detecting rail bolts according to claim 2, characterized in that: The method of preliminarily determining the ROI region where defects exist in the input image according to the extracted features includes: extracting the ROI region where defects exist in the input image through the RPN network.

7. The method for detecting rail bolts according to claim 1 or 2, characterized in that: The method for obtaining the real image of each rail bolt to be inspected on the production line includes: performing image enhancement, smoothing and denoising processing on the original image of each rail bolt to be inspected on the production line acquired by the image acquisition device, and segmenting the bolt body area in the image from the background where the bolt is located, so as to obtain the real image of the bolt to be inspected, thereby removing the area corresponding to the background and retaining the area corresponding to the bolt.

8. The method for detecting rail bolts according to claim 1 or 2, characterized in that: Also includes: According to the defect detection results output by the defect detection model, the defect type and defect location of the bolts including the bolts determined to have defects are obtained; The defect type is used to classify the defects of all defective bolts so as to collect statistics on the types of defects that occur in the defective bolts. The defect position is used to locate the defect of each defective bolt respectively, so as to make statistics on the defect positions of the defective bolts.

9. A rail bolt detection system, comprising a processor, characterized in that: The processor is configured to execute a computer program to implement the steps of the rail bolt detection method according to any one of claims 1 to 8.

Images