Machine vision detection method and system for missing railway sleeper support fasteners

Abstract

The present invention relates to the field of rail transit detection technology, and specifically relates to a machine vision detection method and system for the lack of railway sleeper support fasteners, including a data acquisition step, a detection model construction step, an environment judgment step, a model selection step, and a defect detection step. By constructing detection models for different environmental types and selecting the optimal detection model according to the environmental type, the problem of long processing time of a single model in a complex environment in the traditional method is avoided, the detection efficiency is improved, potential safety hazards can be timely feedback, and the problems of long detection response time and low detection efficiency in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit detection, and in particular to a machine vision detection method and system for the lack of railway sleeper support fasteners. Background Art

[0002] With the rapid development of rail transit, track detection technology has become an important guarantee for ensuring the safe operation of trains. In recent years, vision-based object recognition methods have been widely used in the field of track detection. They collect track images in real time through on-vehicle cameras and combine image processing technology to identify abnormal objects on the track, such as foreign objects and damage to track components. However, during the high-speed operation of trains, the existing vision recognition methods still face the following technical bottlenecks: (1) Image blurring leads to processing delays: Due to the high train speed, the images collected by on-vehicle cameras are easily affected by motion blur and vibration interference, with low image resolution and blurred edges. Traditional image enhancement and deblurring algorithms require a large amount of computing resources and are difficult to meet the real-time processing requirements. Especially in complex lighting or dynamic environments, the processing time of existing algorithms increases significantly, resulting in lag in object recognition and inability to timely feedback potential safety hazards. (2) Insufficient matching accuracy for blurred images: Existing methods usually rely on a clear image database for feature matching, but it is difficult to extract features from blurred images, and misjudgment or missed detection is likely to occur during direct matching. For example, the contour of the object in the blurred image is deformed or the color is distorted, resulting in inaccurate feature mapping relationships and seriously affecting the reliability of the recognition results. (3) Low efficiency of multi-camera collaborative calibration: To improve the detection coverage rate, existing technologies often use multi-camera systems, but it is difficult to effectively eliminate the geometric deformation and errors of images caused by differences in installation angles or environmental interference between different cameras. Traditional calibration methods rely on complex hardware adjustments or static calibrations and cannot dynamically adapt to the real-time changes during train operation, with deficiencies in both calibration accuracy and efficiency. In addition, there is a lack of targeted correction strategies for the common-mode error and differential-mode error of the multi-camera system, further reducing the robustness of the system.

[0003] Therefore, in order to solve the problems existing in the prior art, the present invention proposes a machine vision detection method and system for the lack of railway sleeper support fasteners. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a machine vision detection method and system for the lack of railway sleeper support fasteners.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] A machine vision detection method for the lack of railway sleeper support fasteners includes the following steps:

[0007] Data acquisition step: Obtain environmental data, train operation data, and image sampling data of sleeper support fasteners. The image sampling data includes several sampling images of support fasteners;

[0008] Detection model construction step: Construct several support fastener missing recognition models according to different detection logics and detection parameters, and calculate the model detection cost of each support fastener missing recognition model. The model detection cost is a comprehensive evaluation index of detection speed and detection accuracy;

[0009] Environment judgment step: Analyze the current detection environment type through an environment judgment model based on the environmental data and train operation data;

[0010] Model selection step: Calculate the correction cost of each support fastener missing recognition model according to the current detection environment type, and calculate the comprehensive cost in combination with the model detection cost. Select the support fastener missing recognition model with the minimum comprehensive cost as the optimal detection model;

[0011] Defect detection step: Input the image sampling data of the sleeper support fasteners into the optimal detection model and output the support fastener missing detection result.

[0012] As a further improvement of the present invention, the support fastener missing recognition model includes a low-brightness environment missing recognition model, a curved track environment missing recognition model, and a straight track missing recognition model. The low-brightness environment missing recognition model completes missing detection by fusing multi-spectral image data and denoised visible light images, and extracting fastener thermal features and geometric distribution rules; the curved track environment missing recognition model completes missing detection by calculating the track curvature to dynamically correct image deformation and matching the dynamic template of the fastener arrangement changing with the curvature; the straight track missing recognition model completes missing detection by identifying the straight arrangement rule of fasteners in the image and combining edge detection and deep learning algorithms.

[0013] As a further improvement of the present invention, the construction of the environment judgment model includes setting the light intensity, track type in the environmental data, and vehicle speed in the train operation data as discrete environmental factors, calculating the environment type judgment value through a supervised learning algorithm according to historical detection data, establishing the mapping relationship between environmental factors and environment types according to the environment type judgment value, and judging and outputting the environment type through a preset judgment value classification threshold. The environmental categories include low-brightness environment, curved track environment, and straight track environment.

[0014] As a further improvement of the present invention, the construction of the low-brightness environment missing recognition model includes registering the visible light image and the infrared image in the image sampling data, and dynamically allocating fusion weights based on the real-time ambient light intensity to complete the fusion of multi-spectral data. By performing a multi-frame superposition noise reduction algorithm on the visible light image to eliminate random noise and enhancing the image contrast through adaptive histogram equalization to complete low-light enhancement, locating the fastener area based on the metal thermal radiation characteristics of the infrared image, and combining with the preset fastener geometric distribution template of the track, calculating the matching degree between the fastener density in the local area and the template. If the matching degree is lower than the threshold and the thermal feature contour is broken, it is output that the support fastener is missing.

[0015] As a further improvement of the present invention, the construction of the curved track environment missing recognition model includes calculating the real-time track curvature data of the track according to the support fastener sampling image, performing polar coordinate transformation on the support fastener sampling image according to the real-time track curvature data, mapping the curved track to obtain a virtual straight line image, generating a dynamic template of the fastener arrangement matching the current real-time track curvature based on a preset curvature fastener spacing mapping table, matching the dynamic template of the fastener arrangement and the actual position of the fastener in the virtual straight line image by the phase correlation method, calculating the deviation value between the adjacent fastener spacing value and the preset ideal spacing value. When the deviation values of the fastener spacing values of three consecutive fasteners exceed the preset threshold, it is output that the support fastener is missing.

[0016] As a further improvement of the present invention, the construction of the straight track missing recognition model includes extracting the fastener edge contour in the support fastener sampling image, detecting the straight arrangement rule of the fastener center points through the Hough transform to generate a fitting straight line of the fastener positions, calculating the lateral offset of each fastener center point from the fitting straight line of the fastener positions. If the lateral offset exceeds the preset offset threshold, the area where the fastener is located is set as an abnormal area, and the local image of the abnormal area is input into a preset convolutional neural network. Using the complete fastener image and the edge detection mask as training data, outputting the fastener missing probability. When the fastener missing probability is greater than the preset probability threshold and the edge detection result is abnormal, it is output that the support fastener is missing.

[0017] As a further improvement of the present invention, the defect detection step includes selecting the image sampling data type and performing image processing according to the data type and data format requirements of the selected optimal detection model, performing size reduction, noise elimination, and equalizing the gray level distribution on the image sampling data to complete the standardization process, inputting the processed image sampling data into the corresponding support fastener missing recognition model, and performing spatio-temporal consistency analysis on the missing areas detected in multiple consecutive frames of the image sampling data. If the same position is determined to be missing in multiple frames, the final detection result is output as the support fastener is missing.

[0018] As a further improvement of the present invention, the calculation of the model detection cost includes calculating a detection speed score based on the relationship between the single detection time of the model and the preset maximum allowable time and minimum theoretical time. The detection speed score represents the time duration. The recall rate and precision rate of the model are statistically calculated through the test data set, and the comprehensive precision score is calculated according to the weight distribution rule that the recall rate is prioritized and the precision rate is secondary. The speed score and the precision score are weighted and summed according to the preset weight, and the model detection cost is generated in combination with the type of the support fastener missing recognition model.

[0019] As a further improvement of the present invention, the calculation of the correction cost includes dynamically adjusting the parameters of the support fastener missing recognition model according to the current environment type, calculating the change range of the model detection accuracy before and after the adjustment, statistically calculating the proportion of the extra time consumed by the model parameter adjustment in the total single detection time, and dynamically allocating weights to the parameter adjustment cost and the real-time cost according to the environmental complexity to generate the final correction cost.

[0020] A machine vision detection system for missing railway sleeper support fasteners, comprising:

[0021] A data acquisition module that acquires environmental data, train operation data, and image sampling data of the sleeper support fasteners. The image sampling data includes a plurality of support fastener sampling images;

[0022] A detection model construction module that constructs a plurality of support fastener missing recognition models according to different detection logics and detection parameters, and calculates the model detection cost of each support fastener missing recognition model. The model detection cost is a comprehensive evaluation index of detection speed and detection accuracy;

[0023] An environment judgment module that analyzes the current detection environment type through an environment judgment model according to the environmental data and train operation data; <I

[0024] A model selection module that calculates the correction cost of each support fastener missing recognition model according to the current detection environment type, combines the model detection cost to calculate the comprehensive cost, and selects the support fastener missing recognition model with the minimum comprehensive cost as the optimal detection model;

[0025] A defect detection module that inputs the image sampling data of the sleeper support fasteners into the optimal detection model and outputs the detection result of the missing support fastener. <I

[0026] The beneficial effects of the present invention are:

[0027] (1) The detection efficiency of the missing railway sleeper support fasteners is improved. By constructing detection models for different environment types and selecting the optimal detection model according to the environment type, the problem of long processing time of a single model in a complex environment in the traditional method is avoided, the detection efficiency is improved, and potential safety hazards can be timely feedback.

[0028] (2) The detection accuracy is improved. By adopting targeted image processing and feature extraction methods, more accurate data processing is performed on the detection models for different environmental types. For example, multispectral fusion and noise reduction processing in low-light environments, curvature dynamic correction and dynamic template matching in curved track environments, edge detection and deep learning algorithms in straight track environments, etc. effectively solve problems such as image blurring and difficult feature extraction in different environments, improve the detection accuracy, and reduce misjudgment and missed detection.

[0029] (3) Enhance the system robustness: The environmental judgment model can accurately judge the current detection environmental type. The model selection module dynamically adjusts the model parameters according to the environmental type and selects the optimal model, enabling the system to adapt to real-time environmental changes during train operation and enhancing the system robustness. At the same time, the construction of multiple models and the calculation of the comprehensive cost improve the adaptability of the system to different environments and data, further enhancing the system robustness. Description of the Drawings

[0030] Figure 1 is the flowchart of the method of the present invention;

[0031] Figure 2 is the schematic diagram of the detection image of the present invention;

[0032] Figure 3 is the schematic diagram of the detected track area of the present invention;

[0033] Figure 4 is the system block diagram of the present invention;

[0034] Figure 5 is the schematic diagram of the system flow of the present invention. Detailed Embodiments

[0035] The present invention will be further described in detail below in conjunction with the drawings and embodiments. The same components are denoted by the same reference numerals. It should be noted that the terms "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to the directions in the drawings, and the terms "bottom surface" and "top surface", "inner" and "outer" refer to the directions towards or away from the geometric center of a specific component respectively.

[0036] A machine vision detection method for the lack of railway sleeper support fasteners, as Figures 1 to 5 shown, includes the following steps:

[0037] Data acquisition step, obtaining environmental data, train operation data and image sampling data of the sleeper support fasteners, and the image sampling data includes a number of support fastener sampling images;

[0038] In practical applications, the data acquisition step uses high-precision on-vehicle cameras and a variety of sensors, including light sensors and speed sensors, to obtain environmental data, train operation data, and image sampling data of sleeper support fasteners. The light sensor collects light intensity data in real time, the speed sensor obtains the train operation speed data in real time, and the on-vehicle camera collects image sampling data of sleeper support fasteners at a certain frame rate to ensure that the collected data has high accuracy and real-time performance.

[0039] The detection model construction step constructs several missing fastener recognition models according to different detection logics and detection parameters, and calculates the model detection cost of each missing fastener recognition model. The model detection cost is a comprehensive evaluation index of detection speed and detection accuracy.

[0040] The detection model construction step constructs a missing recognition model in low-light environment, a missing recognition model in curved track environment, and a missing recognition model in straight track environment according to different detection logics and parameters. When constructing the missing recognition model in low-light environment, a visible light camera and an infrared camera are used to collect image data at the same time, and the visible light image and the infrared image are registered through an image registration algorithm. The image matching algorithm uses a feature point-based registration method, including SIFT feature point extraction and matching algorithm, to ensure the spatial consistency of the multi-spectral image. Then, according to the real-time ambient light intensity, the fusion weights of the visible light image and the infrared image are dynamically allocated through an adaptive algorithm. For example, when the ambient light intensity is low, the fusion weight of the infrared image is increased to highlight the thermal characteristics of the fasteners; when the ambient light intensity is high, the fusion weight of the visible light image is increased to utilize the rich details of the visible light image. The visible light image is processed by multi-frame superposition denoising, specifically, the continuous multi-frame visible light images are superposed and averaged to eliminate random noise, and then the image contrast is enhanced through an adaptive histogram equalization algorithm to make the fastener features in the image clearer. Based on the metal thermal radiation characteristics of the infrared image, a threshold segmentation algorithm is used to locate the fastener area. The threshold can be determined according to historical data and experiments to determine a suitable threshold range, and then combined with the preset fastener geometric distribution template of the track, the matching degree between the fastener density in the local area and the template is calculated. If the matching degree is lower than the preset threshold and the thermal feature contour is broken, it is determined that the support fastener is missing.

[0041] The environment judgment step analyzes the current detection environment type through an environment judgment model according to the environmental data and the train operation data.

[0042] The environmental judgment step takes light intensity, track types such as straight tracks and curved tracks, and vehicle speed as discrete environmental factors. The historical detection data includes detection results and corresponding environmental types under different environmental factors. Through a supervised learning algorithm, such as a decision tree algorithm, the historical detection data is trained to establish a mapping relationship between environmental factors and environmental types. According to the real-time collected environmental factor data, the environmental type judgment value is calculated and compared with a preset judgment value classification threshold to judge the current environmental type.

[0043] The model selection step calculates the correction cost of each support fastener missing recognition model according to the current detected environmental type, and combines the model detection cost to calculate the comprehensive cost. The support fastener missing recognition model with the minimum comprehensive cost is selected as the optimal detection model;

[0044] The model selection step dynamically adjusts the parameters of each support fastener missing recognition model according to the current environmental type. For example, in a low-brightness environment, adjust the fusion weight, noise reduction parameters, etc. of the low-brightness environment missing recognition model; in a curved track environment, adjust the curvature calculation parameters, dynamic template generation parameters, etc. of the curved track environment missing recognition model. Calculate the change range of the model detection accuracy before and after adjustment, and determine a reasonable evaluation index through experiments and statistics to measure the accuracy change, such as the change in recall rate and precision rate. Statistically calculate the proportion of the extra time consumed by the model parameter adjustment in the total time of a single detection. According to the environmental complexity, such as a higher complexity in the low-brightness environment, a medium complexity in the curved track environment, and a lower complexity in the straight track environment, dynamically allocate the weights of the parameter adjustment cost and the real-time cost to generate the final correction cost. Combine the model detection cost, statistically calculate the single detection time, recall rate, and precision rate of the model through the test data set, calculate the comprehensive accuracy score according to the weight allocation rule of giving priority to the recall rate and then the precision rate, and calculate the comprehensive cost by weighted summing the speed score and the accuracy score according to the preset weight. Select the model with the minimum comprehensive cost as the optimal detection model.

[0045] The defect detection step inputs the image sampling data of the sleeper support fasteners into the optimal detection model and outputs the detection result of support fastener missing.

[0046] The defect detection module processes the image sampling data according to the data type and format requirements of the optimal detection model. For example, if the optimal detection model is a missing recognition model for low-light environments, visible light images and infrared images are selected as input data, and size reduction is performed to crop the images to the size required by the model. Noise is eliminated through the median filtering algorithm, and the gray level distribution is equalized through the gray level equalization algorithm to complete the standardization process. The processed images are input into the model, and spatio-temporal consistency analysis is performed on the missing areas detected in multiple consecutive frames of images. Specifically, the positions and detection results of the missing areas in each frame of the image are recorded. If the same position is determined to be missing in multiple consecutive frames, usually 3 frames, the final detection result is output as the support fastener is missing to reduce false positives caused by accidental factors.

[0047] Specifically, as Figures 1 to 5 shown, the support fastener missing recognition model includes a missing recognition model for low-light environments, a missing recognition model for curved track environments, and a missing recognition model for straight tracks. The missing recognition model for low-light environments completes missing detection by fusing multi-spectral image data and denoised visible light images and extracting the thermal characteristics and geometric distribution rules of the fasteners; the missing recognition model for curved track environments completes missing detection by calculating the dynamic correction of image deformation based on the track curvature and matching the dynamic template of the fastener arrangement changing with the curvature; the missing recognition model for straight tracks completes missing detection by identifying the straight arrangement rule of the fasteners in the image and combining edge detection and deep learning algorithms.

[0048] Among them, the implementation of the missing recognition model for low-light environments includes:

[0049] Image registration: Adopt a feature point-based registration algorithm, such as the SIFT algorithm, to extract feature points in visible light images and infrared images, and through feature point matching and transformation matrix calculation, achieve accurate registration of the two images.

[0050] Multi-spectral fusion: Dynamically adjust the fusion weights of visible light images and infrared images according to the real-time ambient light intensity. When the light intensity is low, increase the fusion weight of the infrared image to highlight the thermal characteristics of the fasteners; when the light intensity gradually increases, gradually increase the fusion weight of the visible light image to utilize the detailed information of the visible light image.

[0051] Noise reduction and enhancement processing: Perform multi-frame stacking noise reduction on visible light images, that is, average the continuously collected multi-frame visible light images to eliminate random noise; then perform adaptive histogram equalization processing, divide the image into several sub-regions, and perform histogram equalization on each sub-region separately to enhance the local contrast of the image.

[0052] Fastener positioning and missing detection: Based on the thermal radiation characteristics of metal fasteners in infrared images, the fastener area is located through a threshold segmentation algorithm; combined with a preset fastener geometric distribution template, the matching degree between the fastener density in the local area and the template is calculated, and at the same time, whether the thermal feature contour is complete is analyzed. If the matching degree is lower than the preset threshold and the thermal feature contour is broken, it is determined that the support fasteners in this area are missing.

[0053] The implementation of the missing recognition model for the curved track environment includes:

[0054] Track curvature calculation: Adopt a curvature calculation method based on image processing. For example, by detecting the curve features of the track edge, using the least squares method to fit the curve, and calculating the real-time curvature data of the track.

[0055] Polar coordinate transformation: According to the real-time track curvature data, perform polar coordinate transformation on the sampled image of the support fasteners, map the curved track image into a virtual straight line image, and eliminate the image deformation caused by the track curvature.

[0056] Dynamic template generation: According to the preset curvature-fastener spacing mapping table, find the fastener spacing corresponding to the current real-time track curvature, and generate a dynamic template of the fastener arrangement that matches the current curvature.

[0057] Template matching and missing detection: Search for the dynamic template of the fastener arrangement in the virtual straight line image through the phase correlation method to determine the actual position of the fasteners; calculate the deviation value between the adjacent fastener spacing value and the preset ideal spacing value. When the spacing deviation values of three consecutive fasteners all exceed the preset threshold, it is determined that the support fasteners in this area are missing.

[0058] The implementation of the missing recognition model for the straight track includes:

[0059] Edge contour extraction: Adopt the Canny edge detection algorithm to extract the edge contour of the sampled image of the support fasteners to obtain the edge contour information of the fasteners.

[0060] Hough transform to detect straight lines: Use the Hough transform algorithm to detect the straight line arrangement rule of the fastener center points in the edge contour image and generate a fitting straight line for the fastener positions.

[0061] Lateral offset calculation: Calculate the lateral offset of each fastener center point from the fitting straight line of the fastener positions. If the offset exceeds the preset threshold, mark the area where the fastener is located as an abnormal area.

[0062] Deep learning detection: Input the local image of the abnormal area into a preset convolutional neural network. This network uses a large number of complete fastener images and edge detection masks as training data and outputs the fastener missing probability. When the fastener missing probability is greater than the preset probability threshold and the edge detection result shows that the edge is incomplete, it is determined that the support fasteners in this area are missing.

[0063] Specifically, as Figures 1 to 5 shown, the construction of the environment judgment model includes setting the light intensity, track type in the environment data, and vehicle speed in the train operation data as discrete environment factors, calculating the environment type judgment value through a supervised learning algorithm based on historical detection data, establishing a mapping relationship between the environment factors and the environment type according to the environment type judgment value, and judging and outputting the environment type through a preset judgment value classification threshold. The environment categories include low-light environment, curved track environment, and straight track environment.

[0064] The construction of the environment judgment model uses the decision tree algorithm in supervised learning. First, collect a large amount of historical detection data, including environment data and corresponding environment types under different light intensities, track types, and vehicle speeds, namely low-light environment, curved track environment, and straight track environment. Then, use the light intensity, track type, and vehicle speed as input features, and the environment type as the output label to train the decision tree model. By continuously adjusting the parameters and structure of the decision tree, the model can accurately calculate the environment type judgment value based on the input environment factors and establish a mapping relationship between the environment factors and the environment type. Finally, set a reasonable judgment value classification threshold. For example, when the environment type judgment value is less than 0.3, it is determined as a low-light environment; when it is between 0.3 and 0.7, it is determined as a curved track environment; when it is greater than 0.7, it is determined as a straight track environment, so as to accurately judge the current detected environment type.

[0065] Specifically, as Figures 1 to 5 shown, the construction of the low-light environment missing recognition model includes registering the visible light image and the infrared image in the image sampling data, dynamically allocating fusion weights based on the real-time environmental light intensity to complete the fusion of multi-spectral data, eliminating random noise by performing a multi-frame superposition noise reduction algorithm on the visible light image and enhancing the image contrast through adaptive histogram equalization to complete low-light enhancement, positioning the fastener area based on the metal thermal radiation characteristics of the infrared image, combining the preset fastener geometric distribution template of the track, calculating the matching degree between the fastener density in the local area and the template, and if the matching degree is lower than the threshold and the thermal feature contour is broken, output that the support fastener is missing.

[0066] The specific implementation of model construction includes:

[0067] Multi - spectral image registration and fusion. Using a registration algorithm based on feature - point matching such as the SIFT algorithm, scale - invariant feature points are extracted from visible - light images and infrared images. By two - way matching, homologous points are screened out. After removing mis - matched points using the RANSAC algorithm, the homography matrix is calculated to complete image registration, ensuring that the positions of fasteners in the two types of images are aligned. An ambient light intensity is established, and the unit of ambient light intensity is lux. A mapping table between Lux and the fusion weight is established (for example, when the light intensity < 10Lux, the weight of the infrared image accounts for 70% and the visible light accounts for 30%; when 10Lux ≤ light intensity < 30Lux, the infrared accounts for 50% and the visible light accounts for 50%; when the light intensity ≥ 30Lux, the infrared accounts for 30% and the visible light accounts for 70%). The weight is dynamically adjusted through real - time light - intensity sensor data to balance the thermal features and visible - light details.

[0068] Pre - processing of visible - light images. For multi - frame superposition noise reduction, 5 consecutive visible - light images are collected. Superposition noise reduction is performed by calculating the pixel mean. The image is divided into 8×8 - pixel sub - blocks, and histogram equalization is performed on each sub - block, and the contrast gain is limited (the threshold is set to 2.0) to avoid noise amplification and improve the image contrast in low - light environments.

[0069] Fastener positioning and missing detection. After performing Gaussian blur (kernel size 3×3) on the infrared image, the high - temperature area (the thermal - radiation feature of metal fasteners) is extracted through Otsu threshold segmentation to generate a fastener candidate area mask. A preset standard fastener arrangement template is set (such as the horizontal spacing is 600mm ± 5mm and the vertical spacing is 500mm ± 5mm). The local fastener density (the number of fasteners per unit area) is calculated within the candidate area. If the density is less than 80% of the template density and the thermal - feature contour has a break of ≥ 2 pixels, it is determined that the fastener is missing.

[0070] Specifically, as Figures 1 to 5 shown, the construction of the missing - recognition model for the curved - track environment includes calculating the real - time track curvature data of the track based on the sampled image of the support fastener. The real - time track curvature calculation includes:

[0071] Performing Canny edge detection on the sampled image of the support fastener to extract the track - edge curve; using the least

[0072] squares method to fit a quadratic curve , and calculating the curvature formula: , taking the mean curvature of 5 points in the central area of the track

[0073] as the real - time curvature data. Performing polar - coordinate transformation on the sampled image of the support fastener according to the real - time track curvature data to map the curved track to a virtual straight - line image. The polar - coordinate transformation for correcting deformation includes taking the center of the track curvature as the pole and converting the curved - track image from the Cartesian coordinate system to the polar coordinate system. The formula is: , where (x0, y0) are the coordinates of the center of curvature. After transformation, a virtual straight track image is generated to eliminate the effects of bending deformation on fastener spacing. Based on a preset curvature-fastener spacing mapping table, a dynamic fastener layout template matching the current real-time track curvature is generated. Phase correlation is used to match the dynamic fastener layout template with the actual fastener positions in the virtual straight track image. The deviation between the spacing between adjacent fasteners and the preset ideal spacing is calculated. Dynamic template generation and matching is then performed. Based on the curvature spacing mapping table (for example, a curvature of k = 0.001 / mm corresponds to a spacing of 580 mm, and a curvature of k = 0.002 / mm corresponds to a spacing of 560 mm), a dynamic fastener layout template with equal spacing is generated. The template length covers three fastener spacings. In the virtual straight track image, a phase correlation algorithm is used to calculate the translation between the template and the actual fastener image to locate the fastener center. The deviation between adjacent fastener spacings is calculated. If three consecutive spacing deviations exceed ±15 mm (a preset threshold), a fastener is considered missing. When the deviation between the fastener spacing values of three consecutive fasteners and the preset ideal spacing value exceeds the preset threshold, the output is that the support fastener is missing.

[0074] Specifically, such as Figures 1 to 5 As shown, the construction of the linear track missing recognition model includes extracting the fastener edge contour in the support fastener sampling image, detecting the linear arrangement pattern of the fastener center points through Hough transform, generating a fastener position fitting line, and calculating the lateral offset between each fastener center point and the fastener position fitting line. If the lateral offset exceeds a preset offset threshold, the area where the fastener is located is set as an abnormal area, and the local image of the abnormal area is input into a preset convolutional neural network. The fastener complete image and edge detection mask are used as training data to output the fastener missing probability. When the fastener missing probability is greater than the preset probability threshold and the edge detection result is abnormal, the output is that the support fastener is missing.

[0075] Edge contour extraction and line detection include using the Canny edge detection algorithm (Gaussian kernel size 5×5, high and low threshold ratio 3:1) to extract the fastener edge contour. Hough line detection is performed on the edge image, setting the polar angle range [-10°, 10°] (corresponding to the linear track direction), screening the line where the fastener center point is located, and generating the fitted line equation y=mx+b. For each fastener center point (x i ,y i ), calculate its vertical distance to the fitting line, and if the distance exceeds the preset threshold, mark the 30×30 pixel area where the fastener is located as an abnormal area.

[0076] Using the ResNet-18 lightweight model, the input is an abnormal area image (size 30×30 pixels, grayscale image), and the output is the probability of fastener missing (0-1). 100,000 complete fastener images (including edge detection masks) and 20,000 missing images are collected, and the data is enhanced by random rotation (±10°) and scaling (±5%); the loss function uses binary cross-entropy, and the optimizer is Adam (learning rate 1e -4 ). When the CNN output probability > 0.9 and the edge detection mask shows that the edge break length > 10 pixels, it is determined that the fastener is missing.

[0077] Specifically, as Figures 1 to 5 shown, the defect detection steps include, according to the data type and data format requirements of the selected optimal detection model, selecting the image sampling data type and performing image processing, performing size cropping, noise elimination, and equalizing the gray level distribution on the image sampling data to complete the standardization process, inputting the processed image sampling data into the corresponding support fastener missing recognition model, performing spatio-temporal consistency analysis on the missing areas detected in multiple consecutive frames of the image sampling data. If the same position is determined to be missing in multiple frames, the final detection result is output as the support fastener missing.

[0078] Image standardization process

[0079] Size cropping: According to the input requirements of the optimal model, the original image is cropped to 512×512 pixels (retaining the central field of view of the fastener area).

[0080] Noise elimination: Median filtering (kernel size 3×3) is used to remove salt-and-pepper noise, and then Gaussian filtering (kernel size 5×5, standard deviation 1.5) is used to smooth the image.

[0081] Gray level equalization: Global histogram equalization is performed on the image to expand the gray level value range to [0, 255] and improve the overall contrast.

[0082] Spatio-temporal consistency analysis

[0083] Multi-frame detection: 10 consecutive frames of images are collected, and the coordinates of the missing areas detected in each frame are recorded.

[0084] Consistency judgment: If the same coordinate area is determined to be missing in ≥6 frames (time window is about 0.2 seconds, assuming a frame rate of 30fps), then it is confirmed that this area is a real missing, and the detection result is output; otherwise, it is regarded as a false detection and excluded.

[0085] Specifically, as Figures 1 to 5As shown, the calculation of the model detection cost includes calculating a detection speed score based on the relationship between the single - detection time of the model and the preset maximum allowable time and minimum theoretical time. The detection speed score represents the time duration. The recall rate and precision rate of the model are statistically calculated through the test data set, and a comprehensive precision score is calculated according to the weight assignment rule of giving priority to recall rate and then precision rate. The speed score and the precision score are weighted and summed according to the preset weights, and the model detection cost is generated in combination with the type of the support fastener missing recognition model.

[0086] The model detection cost is generated by comprehensively evaluating the detection speed and precision: first, calculate the speed score based on the single - detection time of the model and the preset time interval, then statistically calculate the recall rate and precision rate through the test data and calculate the comprehensive precision score according to the weights, and finally, weight and sum the speed score and the precision score according to the preset weights, and generate the final model detection cost in combination with the model type. The smaller the value, the better the comprehensive performance of the model.

[0087] The calculation configuration of the model detection cost is as follows: ;

[0088] Among them, C model is the model detection cost, which is a cost index that comprehensively measures the detection speed and precision. The smaller the value, the better the model performance; t is a time parameter representing the single - detection time, and its value range is [t min , t max , where t min is the minimum theoretical time, t max is the maximum allowable time, α is a quantization parameter of the environmental type, calculated by α = 1+log(1 + EnvScore), EnvScore ∈ [0,10] is the normalized value of the environmental sensor data, g(Prec, Rec) is the precision comprehensive function, Prec is the precision rate, Rec is the recall rate, and the precision comprehensive function is a non - linear combination of the precision rate and the recall rate, g = Rec·tanh(γ·Prec), where γ is the calibration coefficient with a default value of 2, β is the speed penalty coefficient representing the exponential decay weight when detecting the time consumption,

[0089] is the model sensitivity term, representing the sum of the squares of the gradients of the model F with respect to the input image I k and the parameter θ, N is the number of test images,

[0090] is a tiny constant used to ensure that the denominator is not zero, F(I , θ) is the detection model output function, which is the inference result of the deep - learning model, I k is the k - th frame image, and θ is the model parameter, k which is used to suppress timeout detection, and the cost increases exponentially as the time consumption gets longer. Used to suppress timeout detection, the longer the time consumption, the exponential growth of the cost.

[0091] Specifically, as Figures 1 to 5 shown, the calculation of the correction cost includes dynamically adjusting the parameters of the support fastener missing recognition model according to the current environment type, calculating the change range of the model detection accuracy before and after the adjustment, counting the proportion of the extra time consumed by the model parameter adjustment in the total time of a single detection, dynamically allocating weights to the parameter adjustment cost and the real-time cost according to the environmental complexity, and generating the final correction cost.

[0092] The correction cost is generated based on the impact of dynamically adjusting the model parameters according to the current environment type: first, calculate the change range of the model detection accuracy before and after the parameter adjustment, then count the proportion of the parameter adjustment time in the total time of a single detection, and then dynamically allocate the weights of the accuracy change cost and the real-time cost according to the environmental complexity (such as high complexity for low brightness / bent track and low complexity for straight track), and finally generate the correction cost by weighting, reflecting the requirements of the environment for the model adaptability.

[0093] The correction cost calculation is configured with: ;

[0094] Among them, C adj is the correction cost, is the comprehensive cost of model parameter adjustment, is the environmental parameter domain, representing the integral domain of multi-dimensional environmental factors, , where L light is the maximum light intensity, K curve is the maximum track curvature, V speed is the maximum vehicle speed, is the balance coefficient between environmental complexity and real-time performance, =erf(ρ·EnvScore), erf is the error function, ρ is the scaling factor, ∇ θ ΔA is the accuracy gradient change, representing the gradient difference of the model detection accuracy before and after the parameter adjustment, calculated by automatic differentiation, is the proportion of adjustment time, which is the ratio of the parameter adjustment time to the total time of a single detection, is the real-time sensitivity coefficient, which is a preset parameter, is the environmental non-linear mapping function, used to non-linearly map environmental factors to a high-dimensional space, is the model curvature penalty term, representing the sum of the squares of the second-order derivatives of the loss function L with respect to the parameter θ, reflecting the convergence stability of the model.

[0095] A machine vision detection system for missing support fasteners of railway sleepers, as Figures 1 to 5 shown, includes:

[0096] A data acquisition module, which acquires environmental data, train operation data, and image sampling data of the sleeper support fasteners. The image sampling data includes several support fastener sampling images;

[0097] The detection model construction module constructs several support fastener missing recognition models according to different detection logics and detection parameters, and calculates the model detection cost of each support fastener missing recognition model. The model detection cost is a comprehensive evaluation index of detection speed and detection accuracy;

[0098] The environment judgment module analyzes the current detection environment type through the environment judgment model according to the environment data and train operation data;

[0099] The model selection module calculates the correction cost of each support fastener missing recognition model according to the current detection environment type, combines the model detection cost to calculate the comprehensive cost, and selects the support fastener missing recognition model with the minimum comprehensive cost as the optimal detection model;

[0100] The defect detection module inputs the image sampling data of the sleeper support fastener into the optimal detection model and outputs the detection result of the support fastener missing.

[0101] The machine vision detection system for railway sleeper support fastener missing is built based on an on-vehicle distributed computing platform and includes the following core hardware modules:

[0102] Data acquisition module:

[0103] Image acquisition unit: A line array camera (resolution ≥ 5000×1000 pixels, frame rate ≥ 100fps) installed at the bottom of the train, equipped with infrared and visible light dual-band lenses, supporting synchronous acquisition of multi-spectral images; auxiliary cameras are installed on both sides of the train to cover the fastener vision on both sides of the track.

[0104] Environment perception unit: Integrated light intensity sensor (measurement range 0 - 10000Lux, accuracy ±5%), vehicle speed sensor (CAN bus access to the train operation control system), track curvature sensor (based on inertial navigation and vision fusion algorithm).

[0105] Detection model construction module:

[0106] Edge computing server: Equipped with NVIDIA Jetson Xavier NX, built-in CUDA acceleration unit, supporting parallel construction of multiple models (low brightness, curved track, straight track models), and optimizing the model inference speed through TensorRT.

[0107] Environment judgment module:

[0108] Embedded processor: An ARM Cortex-A72 architecture chip, pre-deployed with an environment judgment model based on decision tree, real-time processing of data such as light intensity, vehicle speed, track type, etc., and outputting the environment type (low brightness / curved track / straight track).

[0109] Model selection module:

[0110] Real-time scheduler: Implemented on FPGA for high-speed logic control, dynamically switches the optimal model according to the environment type and model cost (detection cost + correction cost), with a switching delay ≤ 10 ms.

[0111] Defect detection module:

[0112] Image preprocessing unit: Integrates a dedicated image processing chip (such as Xilinx Zynq), supports parallel execution of standardized processing such as size cropping, noise reduction, and grayscale equalization;

[0113] Detection result output unit: Transmits the detection results (including missing position coordinates and confidence levels) to the train monitoring system through an Ethernet interface and triggers an on-vehicle alarm device (such as an audible and visual alarm).

[0114] System working process

[0115] Step 1: Data synchronization and acquisition

[0116] The line array camera acquires track images (including visible and infrared bands) at a frame rate of 100 fps. The light intensity sensor outputs the ambient light value in real time. The vehicle speed sensor obtains the train running speed (accuracy ±0.1 km / h) through the CAN bus. The track curvature sensor calculates the track curvature in real time through a vision algorithm (updated every 50 ms).

[0117] The data of each module is calibrated with a time stamp through a synchronous clock (accuracy ±1 μs) to ensure strict alignment of the image data and the environment data.

[0118] Step 2: Dynamic construction of the detection model

[0119] At the initial stage of train startup (or during regular maintenance), the edge computing server offline trains three types of detection models based on historical data (including fastener images under different lighting and track types):

[0120] Low-brightness model: Fuses multi-spectral image features and uses TripletLoss to optimize the joint expression of thermal features and geometric features during training;

[0121] Curved track model: Based on the track image dataset with different curvatures, trains a dynamic template generation algorithm;

[0122] Straight track model: Trains a CNN network using a large-scale straight track fastener image dataset (including complete and missing samples).

[0123] The server calculates the detection cost of each model in real time (comprehensive evaluation of speed and accuracy) and stores it in the model parameter library.

[0124] Step 3: Real-time Judgment of Environment Type

[0125] The embedded processor receives the data of light intensity (L), vehicle speed (V), and track curvature (k), and judges the environment type according to the following rules:

[0126] Low-brightness environment: L < 30 Lux and V ≤ 80 km / h;

[0127] Curved track environment: k > 0.0005 / mm or the track type is marked as "curve section";

[0128] Straight track environment: Other situations.

[0129] The environment judgment result is output to the model selection module in the form of one-hot encoding.

[0130] Step 4: Dynamic Selection of the Optimal Model

[0131] The real-time scheduler pre-adjusts the parameters of each model according to the environment type (such as preloading high-infrared weight parameters for the low-brightness model), and calculates the correction cost (reflecting the impact of parameter adjustment on accuracy and speed);

[0132] Combining the detection cost and the correction cost, select the model with the minimum cost as the currently executed model (for example: in a low-brightness environment, preferentially select the low-brightness model, and if its cost is higher than the cost of the curved track model + correction cost, dynamically switch).

[0133] Step 5: Defect Detection and Result Output

[0134] The image preprocessing unit performs standardization processing on the input image: crop it to the fastener ROI area (512×512 pixels), reduce noise through median filtering + Gaussian filtering, and then enhance the contrast through histogram equalization;

[0135] The preprocessed image is input into the optimal model, and the missing probability and position coordinates of the fasteners are output;

[0136] The defect detection module performs spatio-temporal consistency analysis on the detection results of 10 consecutive frames (confirmation is made if the same position is detected as missing in ≥6 frames). The final result is transmitted to the train monitoring system through Ethernet, and the track mileage and image screenshots of the missing position are displayed on the on-vehicle terminal.

[0137] 3. System Integration and Verification

[0138] Installation layout: The line array camera is installed in the middle at the bottom of the train bogie, and the optical axis is perpendicular to the track plane; the auxiliary cameras are installed on both sides at a 45° angle to cover the fasteners at the track edge; the sensors and computing devices are integrated in the on-vehicle cabinet, and vibration interference is reduced through shock absorption devices.

[0139] Test verification: In the laboratory environment, use a simulated orbital platform (supporting adjustable lighting and curvature) to verify the detection accuracy of each model (recall rate ≥ 98%, precision rate ≥ 95%); in the actual line test, the average detection delay of the system for missing fasteners ≤ 200 ms, meeting the real-time requirement.

[0140] 4. Differences from the prior art

[0141] Multi-modal fusion architecture: Different from the traditional single-camera vision system, this system improves the feature robustness in complex environments through multi-spectral image fusion and multi-camera collaboration;

[0142] Dynamic model scheduling mechanism: Break through the adaptability limitations of static models to environmental changes, and achieve a dynamic balance between detection efficiency and accuracy through a cost-driven model selection strategy;

[0143] Spatio-temporal consistency verification: Through multi-frame data association analysis, effectively suppress false detections caused by motion blur and random noise, and the false alarm rate is reduced by more than 80% compared with traditional single-frame detection.

[0144] The above shows and describes the basic features, principles and advantages of the present invention. It should be noted that the present invention is not limited by the above embodiments, but only some embodiments. Without departing from the spirit and scope of the present invention, several improvements and supplements made are considered within the protection scope of the present invention.

Claims

1. A machine vision detection method for the absence of railway sleeper support fasteners, characterized in that, It includes the following steps: A data acquisition step of obtaining environmental data, train operation data, and image sampling data of sleeper support fasteners, where the image sampling data includes several support fastener sampling images; A detection model construction step of constructing several support fastener missing recognition models according to different detection logics and detection parameters, and calculating the model detection cost of each support fastener missing recognition model, where the model detection cost is a comprehensive evaluation index of detection speed and detection accuracy; An environment judgment step of analyzing the current detection environment type through an environment judgment model based on the environmental data and train operation data; A model selection step of calculating the correction cost of each support fastener missing recognition model according to the current detection environment type, and calculating the comprehensive cost in combination with the model detection cost, and selecting the support fastener missing recognition model with the minimum comprehensive cost as the optimal detection model; A defect detection step of inputting the image sampling data of the sleeper support fasteners into the optimal detection model and outputting the support fastener missing detection result; The support fastener missing recognition model includes a low-brightness environment missing recognition model, a curved track environment missing recognition model, and a straight track missing recognition model. The low-brightness environment missing recognition model completes missing detection by fusing multi-spectral image data and denoised visible light images and extracting fastener thermal characteristics and geometric distribution rules; the curved track environment missing recognition model completes missing detection by calculating the track curvature to dynamically correct image deformation and matching the dynamic template of the fastener arrangement changing with the curvature; the straight track missing recognition model completes missing detection by identifying the straight arrangement rule of the fasteners in the image and combining edge detection and deep learning algorithms; The calculation of the correction cost includes dynamically adjusting the parameters of the support fastener missing recognition model according to the current environment type, calculating the change range of the model detection accuracy before and after the adjustment, counting the proportion of the additional time consumed by the model parameter adjustment in the total time of a single detection, and dynamically allocating weights to the parameter adjustment cost and the real-time cost according to the environmental complexity to generate the final correction cost.

2. The machine vision detection method for the lack of railway sleeper support fasteners according to claim 1, characterized in that, The construction of the environment judgment model includes setting the light intensity, track type in the environmental data, and vehicle speed in the train operation data as discrete environmental factors, calculating the environmental type judgment value through a supervised learning algorithm according to historical detection data, establishing the mapping relationship between the environmental factors and the environmental type according to the environmental type judgment value, and judging and outputting the environmental type through a preset judgment value classification threshold. The environmental categories include low-brightness environment, curved track environment, and straight track environment.

3. The machine vision detection method for the absence of railway sleeper support fasteners according to claim 1, characterized in that, The construction of the low-brightness environment missing recognition model includes registering the visible light image and the infrared image in the image sampling data, and dynamically allocating fusion weights based on the real-time ambient light intensity to complete the fusion of multi-spectral data. The random noise is eliminated by performing a multi-frame superposition noise reduction algorithm on the visible light image, and the image contrast is enhanced by adaptive histogram equalization to complete low-light enhancement. The fastener area is located based on the metal thermal radiation characteristics of the infrared image, and combined with the preset fastener geometric distribution template of the track, the matching degree between the fastener density in the local area and the template is calculated. If the matching degree is lower than the threshold and the thermal feature contour is broken, it is output that the support fastener is missing.

4. The machine vision detection method for the missing railway sleeper support fasteners according to claim 1, characterized in that, The construction of the curved track environment missing recognition model includes calculating the real-time track curvature data of the track according to the support fastener sampling image, performing polar coordinate transformation on the support fastener sampling image according to the real-time track curvature data, mapping the curved track to obtain a virtual straight-line image, generating a dynamic template of fastener arrangement matching the current real-time track curvature based on a preset curvature-fastener spacing mapping table, matching the dynamic template of fastener arrangement and the actual position of the fasteners in the virtual straight-line image by the phase correlation method, calculating the deviation value between the adjacent fastener spacing value and the preset ideal spacing value. When the deviation values of the fastener spacing values of 3 consecutive fasteners exceed the preset threshold, it is output that the support fastener is missing.

5. A machine vision detection method for the absence of railway sleeper support fasteners according to claim 1, characterized in that, The construction of the straight track missing recognition model includes extracting the fastener edge contour in the support fastener sampling image, detecting the straight-line arrangement rule of the fastener center points by the Hough transform, generating a fitting straight line for the fastener positions, calculating the lateral offset of each fastener center point from the fitting straight line for the fastener positions. If the lateral offset exceeds the preset offset threshold, the area where the fastener is located is set as an abnormal area, and the local image of the abnormal area is input into a preset convolutional neural network. Using the complete fastener image and the edge detection mask as training data, the fastener missing probability is output. When the fastener missing probability is greater than the preset probability threshold and the edge detection result is abnormal, it is output that the support fastener is missing.

6. The machine vision detection method for the absence of railway sleeper support fasteners according to claim 1, wherein, The defect detection step includes selecting the image sampling data type and performing image processing according to the data type and data format requirements of the selected optimal detection model, performing size reduction, noise elimination and equalizing the gray level distribution on the image sampling data to complete the standardization process, inputting the processed image sampling data into the corresponding support fastener missing recognition model, and performing spatio-temporal consistency analysis on the missing areas detected in multiple consecutive frames of the image sampling data. If the same position is determined to be missing in multiple frames, the final detection result is output as the support fastener is missing.

7. A machine vision detection method for the absence of railway sleeper support fasteners according to claim 1, characterized in that, The calculation of the model detection cost includes calculating a detection speed score based on the relationship between the single - time detection duration of the model and the preset maximum allowable duration and minimum theoretical duration. The detection speed score represents the duration of time. The recall rate and precision rate of the model are statistically calculated through a test data set, and a comprehensive precision score is calculated according to the weight distribution rule that the recall rate takes precedence and the precision rate takes second place. The speed score and the precision score are weighted and summed according to a preset weight, and the model detection cost is generated in combination with the type of the support fastener missing recognition model.

8. A machine vision detection system for the absence of railway sleeper support fasteners, applicable to the machine vision detection method for the absence of railway sleeper support fasteners described in any one of claims 1 to 7, characterized in that, It includes: A data acquisition module that acquires environmental data, train operation data, and image sampling data of sleeper support fasteners. The image sampling data includes a number of support fastener sampling images; A detection model construction module that constructs a number of support fastener missing recognition models according to different detection logics and detection parameters, and calculates the model detection cost of each support fastener missing recognition model. The model detection cost is a comprehensive evaluation index of detection speed and detection precision; An environment judgment module that analyzes the current detection environment type through an environment judgment model based on the environmental data and train operation data; A model selection module that calculates the corrected cost of each support fastener missing recognition model according to the current detection environment type, calculates the comprehensive cost in combination with the model detection cost, and selects the support fastener missing recognition model with the minimum comprehensive cost as the optimal detection model; A defect detection module that inputs the image sampling data of the sleeper support fasteners into the optimal detection model and outputs the support fastener missing detection result.

Images