A method for detecting missing railway fastenings based on machine vision

Abstract

The present application relates to the technical field of electric digital data processing, in particular to a railway fastener missing detection method based on machine vision. First, the gray scale image of the target railway fastener under each view angle is obtained, and the optimal template image is screened out in the standard template image library based on the gray scale value of each pixel; the consistency of the gray scale image under each view angle is evaluated by analyzing the pixel gray scale value offset between the gray scale image and the optimal template image, and the matching degree is obtained, and the matching weight is adaptively determined based on the overall similarity of the gray scale image and the optimal template, so as to ensure that the high-reliability image occupies a higher weight in the fusion; finally, the matching degree and the matching weight are fused to obtain the comprehensive matching value to distinguish the complete state and the missing state of the fastener. The present application solves the misjudgment and missed judgment problem caused by the single template angle of the traditional method, and improves the accuracy of the detection result.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and specifically to a method for detecting missing railway fasteners based on machine vision. Background Technology

[0002] Railway fasteners are critical connectors that firmly fix rails to sleepers. They are essential components for securing the rails and sleepers, and their absence or incompleteness can easily lead to track safety hazards. Current methods for detecting missing railway fasteners primarily rely on visible light image processing and template matching. This involves acquiring optical images of the track and calculating the difference between the image of the fastener in the test section and a standard template; the greater the difference, the higher the probability of a missing fastener. Existing technologies typically match the acquired image with a single fixed template during the template matching process, analyzing the matching results to determine the extent of fastener loss.

[0003] However, when existing detection methods match the acquired images with template images, if the fastener structure is complex, the images taken from different angles will show different structural features. In this case, an image from a single angle cannot fully reflect the fastener structure, and therefore cannot fully reflect the missing fastener situation. If the acquired image to be tested is forcibly matched with a complete, unmissing fastener taken from a single fixed angle, errors in the matching results caused by the different shooting angles during image acquisition will be introduced, thus affecting the accuracy of the matching and leading to false positives and false negatives in fastener missing detection. Therefore, using only image templates from fixed viewing angles for image matching results in low accuracy in fastener missing detection. Summary of the Invention

[0004] To address the problem that existing technologies using only a single fixed-view image template for image matching result in low accuracy in detecting missing railway fasteners, this invention aims to provide a machine vision-based method for detecting missing railway fasteners. The specific technical solution adopted is as follows: Acquire grayscale images of the target railway fastener from each viewpoint; the grayscale images include the grayscale value and pixel coordinates of each pixel; acquire a standard template image library, which is based on railway fasteners without missing data from all viewpoints; Based on the positional change trend of the pixel coordinates of all pixels representing the same real-world meaning in the grayscale image, a standard template image library is used to filter and determine the corresponding optimal template image; based on the relative offset between the grayscale values ​​of each pixel in the grayscale image and the corresponding optimal template image, the corresponding matching degree is determined; based on the overall similarity between the grayscale image and the corresponding optimal template image, the corresponding matching weight is analyzed and determined. Based on the matching weight, all the matching degrees are fused to determine a comprehensive matching value; based on the comprehensive matching value, railway fastener missing detection is performed.

[0005] Furthermore, the method for obtaining the optimal template image includes: Extract the corresponding fastener region from each grayscale image; perform line fitting on all pixels in each grayscale image to construct a rail fitting line and a sleeper fitting line respectively; determine the center coordinates of each grayscale image based on the intersection between the rail fitting line and the sleeper fitting line. In the fastener region of each grayscale image, the template distance between each grayscale image and each standard template image in the standard template image library is obtained based on the spatial distance between the center coordinates of each grayscale image and the center coordinates of each standard template image in the standard template image library. The standard template image corresponding to the minimum value of the template distance is used as the optimal template image for each grayscale image.

[0006] Furthermore, the method for obtaining the matching degree includes: Based on the relative spatial position between the center coordinates of each grayscale image and the corresponding optimal template image, each grayscale image is aligned to determine the image to be tested; Based on the pixel grayscale difference between each pixel in the preset fastener pixel area of ​​each test image and the corresponding optimal template image in the preset fastener pixel area, the matching reference degree of each pixel in the preset fastener pixel area of ​​each test image is obtained. The matching degree of each grayscale image is calculated based on the overall size of the matching reference degree of all pixels in the preset fastener pixel area of ​​each image to be tested.

[0007] Furthermore, the method for obtaining the matching weight includes: The degree of similarity between each grayscale image and the center coordinates of the optimal template image is determined based on the relative deviation between the center coordinates of each grayscale image and the center coordinates of the optimal template image. The imaging quality of each grayscale image is determined based on the discreteness of all pixels in the preset fastener pixel area of ​​each grayscale image; The proximity and the imaging quality are weighted and fused to determine the matching weight value for each grayscale image; The matching weight of each grayscale image is determined based on the proportion of each matching weight value among all matching weight values.

[0008] Furthermore, the method for obtaining the comprehensive matching value includes: Based on the matching weight and corresponding matching degree of each grayscale image, a reference matching value for each grayscale image is calculated; the reference matching value is positively correlated with the matching weight and the matching degree, respectively. By combining all the aforementioned reference matching values, a comprehensive matching value is determined.

[0009] Furthermore, the process of detecting missing railway fasteners includes: If the overall matching value is greater than the preset integrity threshold, the railway fastener is determined to be in an intact state. If the overall matching value is less than or equal to the preset integrity threshold, the railway fastener is determined to be in a missing state.

[0010] Furthermore, the absolute value of the difference is used to calculate the pixel grayscale difference between the grayscale value of each pixel in the preset fastener pixel region of each image to be tested and the grayscale value of each pixel in the corresponding optimal template image in the preset fastener pixel region.

[0011] Furthermore, the method for obtaining the degree of proximity includes: By performing a negative correlation mapping between the Euclidean distance of the center coordinates of each grayscale image and the center coordinates of the optimal template image, the degree of closeness of each grayscale image can be obtained.

[0012] Furthermore, the method for acquiring the image quality includes: Each grayscale image is convolved with the Laplacian operator to determine each convolved image; The variance of all pixels in the preset fastener pixel region of each convolutional image is normalized to obtain the imaging quality of each grayscale image.

[0013] Furthermore, the preset integrity threshold is set to 0.8.

[0014] The present invention has the following beneficial effects: This invention first overcomes the problem that images captured from a single angle cannot fully reflect the fastener structure by acquiring grayscale images of the target railway fastener from each viewpoint, providing a reliable analytical basis for subsequent multi-angle fastener detection. Secondly, by selecting the optimal template image that is closest to the grayscale image from each angle, matching deviations caused by viewpoint changes are effectively eliminated, improving the accuracy of template matching. By analyzing the offset of pixel grayscale values ​​to calculate the matching degree, the consistency of grayscale images from each viewpoint can be quantitatively reflected. Simultaneously, based on the similarity between the grayscale image and the optimal template image, the corresponding matching weight is adaptively determined, effectively suppressing interference from imaging noise and ensuring that highly reliable images occupy a higher weight in subsequent analysis, improving the stability of the judgment. Finally, by fusing the matching degree and matching weight, multi-view matching information is fully utilized, significantly eliminating the instability of a single viewpoint, and performing railway fastener missing detection. This objectively reflects the overall matching degree between the fastener and the standard template, effectively distinguishing between complete and missing states, solving the detection misjudgment and missed judgment problems caused by the single template angle in traditional methods, and improving the accuracy of the detection results. Attached Figure Description

[0015] Figure 1 A flowchart of a machine vision-based method for detecting missing railway fasteners provided by the present invention; Figure 2 A schematic diagram of a railway fastener structure provided by the present invention; Figure 3 A front view of a railway fastener provided by the present invention; Figure 4 A flowchart of an optimal template image acquisition method provided by the present invention. Detailed Implementation

[0016] The following description, in conjunction with the accompanying drawings, details a specific scheme for a machine vision-based method for detecting missing railway fasteners provided by the present invention.

[0017] Please see Figure 1 The diagram illustrates a flowchart of a machine vision-based method for detecting missing railway fasteners according to an embodiment of the present invention. The method includes: Step S101: Obtain grayscale images of the target railway fastener from each viewpoint; the grayscale images include the grayscale value and pixel coordinates of each pixel; obtain a standard template image library, which is based on railway fasteners without missing data from all viewpoints.

[0018] like Figure 2 As shown, this is a schematic diagram of a railway fastener structure provided by the present invention. Considering that a railway fastener is a three-dimensional object in reality, the relative position of the camera to the railway fastener will affect the appearance of the railway fastener in the image, and the display state will be different depending on the viewing angle. For example, as shown... Figure 3 As shown, this invention provides a frontal schematic diagram of a railway fastener. The center is an I-shaped rail, with railway fasteners fastened to the rail on either side. Images taken from one side of the fastener typically cannot encompass the other side. If there are gaps in the other side image, the missing portion will be obscured by other fastener parts in the image due to the different camera angles. Therefore, railway fastener images obtained from different perspectives differ. To obtain comprehensive, multi-view images of the railway fastener, this embodiment first acquires grayscale images of the target railway fastener from each perspective, providing a reliable data foundation for subsequent analysis.

[0019] As an example, in one specific implementation of the present invention, an inspection vehicle equipped with a high-definition camera moves at a constant speed along the railway track to take pictures. When the inspection vehicle head first reaches the fastener position, the first picture is triggered. Then, when the inspection vehicle head moves to the center position of the fastener, the second picture is triggered. When the inspection vehicle head leaves the fastener position, the third picture is triggered. All the images taken during this period are used as color images of the target railway fastener from each viewpoint.

[0020] In this embodiment, for example, a laser rangefinder sensor (projecting towards the rail) can be installed on the inspection vehicle head to detect the distance between the inspection vehicle head and the rail in real time. When the first detected change in the distance between the inspection vehicle head and the rail (the change in distance can be the absolute value of the difference between the real-time collected distance between the inspection vehicle head and the rail and the distance collected at the previous sampling time) is greater than a preset thickness threshold (which can be obtained by measuring the minimum distance between the fastener and the rail in advance on the fastener already installed on the rail), the first image is triggered. Afterwards, based on the moving speed of the inspection vehicle and the length of the target railway fastener (usually the length of the railway fastener is located at...), the system can capture images. In this embodiment, the following is taken: ) Calculate the distance (length) of the inspection vehicle to the center of the target railway fastener. The time at which the first shot is taken is recorded as the movement duration. After the movement duration has elapsed after the first shot, the second shot is triggered. Then, when the distance between the inspection vehicle and the rail is detected to have recovered and is less than or equal to the preset thickness threshold for the first time, the third shot is triggered.

[0021] After acquiring color images of the target railway fastener from three perspectives, the color images are converted to grayscale to obtain grayscale images of the target railway fastener from each perspective. It should be noted that the number of perspectives captured for each target railway fastener can be set according to the specific implementation scenario; for example, capturing grayscale images from four perspectives can be determined by the implementer.

[0022] Furthermore, a Cartesian coordinate system is established in each grayscale image with the bottom left pixel as the origin, the rightward direction of the grayscale image as the positive horizontal axis, and the upward direction of the grayscale image as the positive vertical axis. Each pixel represents a coordinate point. Unless otherwise specified, each grayscale image in the subsequent embodiments of the present invention refers to a grayscale image taken from each viewpoint of the same target railway fastener, and will not be described individually.

[0023] In this embodiment, the standard template image library contains grayscale images of defect-free railway fasteners from all viewpoints, i.e., standard template images. The method for acquiring all standard template images is the same as the method for acquiring grayscale images from each viewpoint described above. It simply involves replacing the three grayscale images of each target railway fastener from each viewpoint acquired using a pre-calibrated method with thirty grayscale images of defect-free fasteners from each viewpoint (this number can be adjusted by the implementer). After acquisition, each standard template image is input into the trained semantic segmentation U-net model to segment the railway fastener region in each standard template image. This region is determined by the relative position of the track and sleepers. Ultimately, each standard template image contains one railway fastener region. It should be noted that training the semantic segmentation U-net model is a technique well-known to those skilled in the art and requires no further elaboration or limitation here.

[0024] Step S102: Based on the position change trend of the pixel coordinates of all pixels representing the same real-world meaning in the grayscale image, filter in the standard template image library to determine the corresponding optimal template image; determine the corresponding matching degree according to the relative offset between the grayscale values ​​of each pixel in the grayscale image and the corresponding optimal template image; analyze and determine the corresponding matching weight based on the overall similarity between the grayscale image and the corresponding optimal template image.

[0025] Because traditional railway fastener inspection uses a single fixed template for matching, it ignores the differences in fastener images acquired from different angles due to inconsistent shooting angles and the complexity of fastener components, resulting in certain errors in the template matching results. Therefore, this embodiment considers selecting from a standard template image library to find the closest standard template image for each grayscale image from each viewpoint, i.e., the optimal template image.

[0026] Based on the above idea, considering that the position of the fastener is different in grayscale images under different shooting angles, for example, in the first image, the pixels representing the fastener are concentrated at the edge of the image, while in the second image, the pixels representing the fastener are concentrated in the middle of the image, therefore, the optimal template image that is closest to the grayscale image can be obtained by filtering from the standard template image library based on the changing trend of the pixel coordinates representing the same real meaning in the grayscale image (in this embodiment, the pixel coordinates of pixels representing the real meaning of rails or sleepers are used).

[0027] It should be noted that if a railway fastener is missing (e.g., a screw), the missing part will expose the rails, sleepers, or soil behind the fastener. In a grayscale image, this manifests as a significant shift in the grayscale value at the missing location compared to the grayscale value when there is no missing part. Conversely, when no parts are missing in the grayscale image, the shift between the grayscale image and the corresponding optimal template image for each pixel is smaller. In summary, by measuring the shift between the grayscale value of each pixel and the corresponding grayscale value in the optimal template image, a preliminary matching degree can be determined. This allows for the quantification of the degree of matching between the grayscale image and the optimal template image from each viewpoint. This matching degree characterizes the probability that the fastener remains intact from that viewpoint.

[0028] This invention takes into account various factors that affect image quality in reality, such as camera shake. Treating a grayscale image with poor quality as the same type of image as one with good quality would introduce unnecessary errors into subsequent detection and analysis, reducing the accuracy of fastener missing detection. Therefore, since all standard template images in the standard template image library are defect-free images captured under ideal conditions, the higher the overall similarity between a grayscale image and the optimal template image, the higher the imaging quality of the corresponding grayscale image. Thus, it should be given higher weight in subsequent analysis, i.e., assigned a higher matching weight.

[0029] Step S103: Based on the matching weight, fuse all the matching degrees to determine the comprehensive matching value; based on the comprehensive matching value, perform railway fastener missing detection.

[0030] Based on the analysis in step S102 above, this embodiment integrates the matching weights and the corresponding matching degrees, that is, it combines the probability that the fastener can maintain its integrity under each viewpoint to construct a comprehensive matching value. This comprehensive matching value also takes into account the influence of multi-view shooting and imaging quality. Based on this, railway fastener missing detection can significantly improve the accuracy and reliability of fastener missing detection while solving the problem of using a single fixed viewpoint image template for image matching.

[0031] In summary, this invention first overcomes the problem that images captured from a single angle cannot fully reflect the fastener structure by acquiring grayscale images of the target railway fastener from each viewpoint, providing a reliable analytical basis for subsequent multi-angle fastener detection. Secondly, by selecting the optimal template image that is closest to the grayscale image from each angle, it effectively eliminates matching deviations caused by viewpoint changes, improving the accuracy of template matching. By analyzing the offset of pixel grayscale values ​​to calculate the matching degree, it quantitatively reflects the consistency of grayscale images from each viewpoint. Simultaneously, based on the similarity between the grayscale image and the optimal template image, it adaptively determines the corresponding matching weight, effectively suppressing interference from imaging noise and ensuring that highly reliable images occupy a higher weight in subsequent analysis, improving the stability of the judgment. Finally, by fusing the matching degree and matching weight, it comprehensively utilizes multi-view matching information, significantly eliminating the instability of a single viewpoint, and performing railway fastener missing detection. It objectively reflects the overall matching degree between the fastener and the standard template, effectively distinguishing between complete and missing states, solving the detection misjudgment and missed judgment problems caused by the single template angle in traditional methods, and improving the accuracy of the detection results.

[0032] Preferably, in some possible implementations of the embodiments of the present invention, the specific implementation process of obtaining the optimal template image of the grayscale image under each viewpoint in step S102 includes: Please refer to Figure 4 The diagram illustrates a flowchart of an optimal template image acquisition method according to an embodiment of the present invention, the method comprising: Step S401: Extract the corresponding fastener region from each grayscale image; perform straight line fitting on all pixels in each grayscale image to construct a rail fitting line and a sleeper fitting line respectively; determine the center coordinates of each grayscale image based on the intersection between the rail fitting line and the sleeper fitting line.

[0033] The fastener structures on the same section of railway track are usually identical (using the same model). However, the background pixels other than the fasteners in the grayscale image, such as soil and weeds, are different in different shooting angles and shooting stages. If they are treated equally in the calculation process, the accuracy of the final detection result will be reduced. Therefore, before performing subsequent calculations and analysis, it is necessary to remove irrelevant background pixels. That is, the grayscale image under each viewpoint is input into the aforementioned trained semantic segmentation U-net model to extract the fastener region in each grayscale image.

[0034] Considering that the closer the position of the railway fastener in the grayscale image is to its position in the standard template image, the more likely the two images were taken from the same viewpoint, and the closer the connection between the two images, meaning that every part in the grayscale image can be found in the corresponding standard template image, rather than some parts being obscured due to different viewpoints, thus preventing defect detection. However, if there are missing parts in the grayscale image, the fastener area in the grayscale image and the standard template image, even if taken from the same viewpoint, will not completely overlap. The intersection of the edges of the rail and sleeper, as the load-bearing base of the fastener, has an absolutely constant physical position in the image, unaffected by whether the fastener is missing. Based on this idea, in this embodiment, pixels representing the rail or sleeper in reality are used to calibrate the position of the railway fastener in the image.

[0035] Therefore, in one specific implementation of this embodiment, the Hough Transform is first used to extract the fitted straight lines of the rail at the two edges of the rail fastened in the middle of the target railway fastener, and the fitted straight lines of the sleepers at the two edges of the sleepers under the fastener, from the grayscale image. The two fitted straight lines of the rail are parallel to each other, the two fitted straight lines of the sleepers are parallel to each other, and the fitted straight lines of the rail and sleepers are perpendicular to each other. Thus, the coordinates of four intersection points between the fitted straight lines of the rail and sleepers can be obtained. The average coordinates of these four intersection points (the average of all horizontal coordinates is used as the new horizontal coordinate, and the average of all vertical coordinates is used as the new vertical coordinate) are used as the center coordinates of each grayscale image; similarly, the center coordinates of each standard template can be obtained. It should be noted that the Hough Transform is a computational method well-known to those skilled in the art and will not be elaborated upon here.

[0036] It should be further noted that if the shooting distance is relatively close, the captured image will only contain the edge of the rail and the edge of the sleeper on one side adjacent to the fastener. Therefore, in other implementations of this invention, a rail fitting line and a sleeper fitting line can also be extracted, and the intersection of these two fitting lines can be used as the center coordinates.

[0037] Step S402: In the fastener region of each grayscale image, based on the spatial distance between the center coordinates of each grayscale image and the center coordinates of each standard template image in the standard template image library, the template distance between each grayscale image and each standard template image is obtained. In this embodiment, the center coordinates of each grayscale image are determined according to the average value of the pixel coordinates of all pixels.

[0038] The closer the center coordinates of a grayscale image are to the center coordinates of each standard template image, the more consistent the shooting angles are. Therefore, this embodiment of the invention obtains the spatial distance between two center coordinates (in this embodiment, Euclidean distance is used to represent the spatial distance between two center coordinates) and determines a template distance, which can vividly and accurately quantify the consistency of the shooting angle between each grayscale image and each standard template image. As an example, in a specific implementation of this embodiment, for any grayscale image, the Euclidean distance between the center coordinates of the grayscale image and the center coordinates of the standard template image is used as the template distance between each grayscale image and each standard template image.

[0039] Step S403: Use the standard template image corresponding to the minimum value of the template distance as the optimal template image for each grayscale image.

[0040] The smaller the template distance, the more consistent the corresponding positions of the railway fasteners in the grayscale image and the standard template image. For example, the railway fasteners are both located in the middle of the two images. Conversely, the larger the template distance, the more inconsistent the corresponding positions of the railway fasteners in the grayscale image and the standard template image. For example, one is on the left side of the image, and the other is on the right side. This indicates that the viewing angles of the grayscale image and the standard template image are inconsistent. In summary, this embodiment selects the optimal template image for each grayscale image by choosing the standard template image corresponding to the smallest template distance from all standard template images. This significantly improves the accuracy and reliability of missing data detection.

[0041] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the matching degree includes: The grayscale images are aligned based on their relative spatial positions with the center coordinates of the corresponding optimal template image to determine the image to be tested. In this embodiment, the grayscale images are aligned based on the difference between the center coordinates of each grayscale image and the center coordinates of the corresponding optimal template image.

[0042] In this invention, the standard template image is constructed on a straight railway track. This means that when the inspection vehicle acquires images on a straight track, the camera always moves in a straight line. However, in the actual process of acquiring the grayscale image of the target railway fastener, when the fastener is on a straight track, the inspection vehicle only moves in a straight line in one direction. At this time, the grayscale image and the standard template image only shift backward within the image, and the fastener in both the grayscale and standard template images faces the same direction. When the inspection vehicle enters a curved section, such as when the front of the vehicle enters the curved track while the rear remains on the straight track, the camera position rotates relative to the straight direction due to the track curvature. This can cause a lateral shift in the fastener's position in the image, requiring appropriate rotation of the captured grayscale image to align the fastener area in the image with the template image. Therefore, in this embodiment, after obtaining the optimal template image for each grayscale image, the grayscale image needs to be further aligned with the corresponding optimal template image.

[0043] Specifically, based on the two fitted rail lines obtained in each grayscale image in step S401, the tilt angles of the two fitted rail lines relative to the positive horizontal axis of the coordinate system of the grayscale image are calculated. The average of the two tilt angles is taken as the orientation angle of the grayscale image (if there is only one fitted rail line, only the tilt angle of that fitted rail line needs to be calculated). Similarly, the orientation angle of the optimal template image can be obtained, denoted as the template angle. The orientation angle of the grayscale image is subtracted from the template angle of the corresponding optimal template image, and the difference is taken as the rotation correction amount of the grayscale image. With the center coordinates of the grayscale image as the rotation center point and the rotation correction amount as the rotation angle, clockwise rotation can initially align the grayscale image and the optimal template image in the direction. Then, the center coordinates of the optimal template image are subtracted from the center coordinates of the grayscale image (the horizontal coordinates are subtracted accordingly, and the vertical coordinates are subtracted accordingly). The new coordinates are taken as the translation correction amount of the grayscale image. The translation correction amount is added to the coordinates of each pixel of the rotated grayscale image to obtain the initially aligned image. Considering that rotation operations can cause coordinates that were originally integers to become decimals, for example, a pixel coordinate (10,10) rotated 15 degrees around the center coordinate (5,5) becomes (9.65,11.23), therefore, after obtaining the initial aligned image, an affine transformation matrix is ​​constructed based on the rotation correction and translation correction. This affine transformation matrix is ​​used to perform bilinear interpolation resampling on the initial aligned image, and zero-value filling is performed on out-of-bounds regions. The output is a test image that is remapped to an integer coordinate grid and whose size is consistent with the optimal template image. The specific implementation process of constructing the affine transformation matrix based on the rotation correction and translation correction and bilinear interpolation resampling is a well-known technique in the art and will not be elaborated here.

[0044] The matching reference degree of each pixel in the preset fastener pixel region of each test image is obtained based on the pixel grayscale difference between the grayscale value of each pixel in the preset fastener pixel region of each test image and the grayscale value of each pixel in the corresponding optimal template image in the preset fastener pixel region. In this embodiment, the matching reference degree of each pixel in the preset fastener pixel region of each test image is obtained based on the absolute value of the difference between the grayscale value of each pixel in the preset fastener pixel region of each test image and the grayscale value of each pixel in the corresponding optimal template image in the preset fastener pixel region.

[0045] During the capture of grayscale images, interference from natural light is introduced. Therefore, performing independent contrast-limited adaptive histogram equalization (CLAHE) on the image under test and the corresponding optimal template image can eliminate the overall grayscale deviation caused by changes in natural lighting. Since actual measurements introduce errors, even after alignment, the shape and number of pixels of the railway fastener region in the image under test will not be completely identical to those in the corresponding optimal template image. Therefore, in this invention, all pixels in a preset fastener pixel region of both the image under test and the optimal template image are compared and analyzed. That is, the absolute value of the difference between two pixels is calculated at each corresponding pixel position, which can effectively eliminate interference caused by actual measurement errors. It should be noted that contrast-limited adaptive histogram equalization is a well-known technique and does not require further elaboration or limitation here.

[0046] In one specific implementation of this invention, the absolute difference between two grayscale values ​​at the same pixel position in the preset fastener pixel region of the test image and the optimal template image is first calculated. The absolute differences of all pixel positions are then normalized by a minimum-maximum value normalization process to obtain each normalized absolute difference value (ranging from [0,1]). Then, a positive integer 1 is subtracted from each normalized absolute difference value (this is because a larger absolute difference value indicates a larger grayscale deviation, suggesting a worse matching situation; therefore, subtracting the normalized absolute difference value from a positive integer 1 ensures that the matching reference value is consistent with the matching situation). The results are positively correlated and used as the matching reference for each corresponding pixel in the image under test. If there are missing parts in the image under test, the background (grass, sleepers, etc.) at the missing location will be exposed. At this time, the absolute difference between the gray values ​​of the two pixels at the missing location will be large, and the matching reference will be small. Conversely, when there are no missing parts in the image under test, the absolute difference between the gray values ​​of the two pixels at the same location will be small, and the matching reference will be large. This can vividly represent the degree of matching between the railway fastener area in the grayscale image and the railway fastener area in the optimal template image.

[0047] In the above-mentioned maximum and minimum value normalization process, the maximum and minimum values ​​used can be obtained from all data in the statistical database to obtain the maximum and minimum values ​​under the dimension to be normalized (in this embodiment, the maximum value is preferably 255 and the minimum value is 0), thereby realizing the normalization of the absolute value of each difference. The specific process is a technical means well known to those skilled in the art, and will not be elaborated or limited here.

[0048] In one specific implementation of this invention, the railway fastener region in the optimal template image corresponding to each image to be tested (the railway fastener region of each standard template image is obtained in step S101) is used as the preset fastener pixel region between each image to be tested and the optimal template image; therefore, for each image to be tested, since the corresponding optimal template image is different, the preset fastener pixel region corresponding to each image to be tested is also different.

[0049] The matching degree of each grayscale image is calculated based on the overall size of the matching reference degree of all pixels in the preset fastener pixel area of ​​each image to be tested.

[0050] In this invention, the average of the matching reference scores of all pixels in the preset fastener pixel region of each test image is used as the matching score of each grayscale image. The higher the matching score, the smaller the difference between the grayscale image and the optimal template image, that is, the better the matching between the grayscale image and the template. The matching score ranges from [0,1].

[0051] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the matching weight includes: The degree of similarity between each grayscale image and the center coordinates of the optimal template image is determined based on the relative deviation between the center coordinates of each grayscale image and the center coordinates of the optimal template image. The imaging quality of each grayscale image is determined based on the discreteness of all pixels in the preset fastener pixel area of ​​each grayscale image; The proximity and the imaging quality are weighted and fused to obtain a matching weight value for each grayscale image; The matching weight of each grayscale image is determined based on the proportion of each matching weight value among all matching weight values.

[0052] The degree of similarity between the grayscale image at each angle and its corresponding optimal template image varies. The greater the similarity, the smaller the error caused by the difference in image shooting angle. Correspondingly, the grayscale image at that angle should have a higher weight in subsequent processes. Since the center coordinates of the fastener reflect the image shooting angle, the smaller the relative deviation between the center coordinates of each grayscale image and the center coordinates of the optimal template image, the closer their shooting angles are.

[0053] In practice, different shooting devices produce different shooting qualities. Less noise and clearer details indicate higher image quality for grayscale images. When compared with the optimal template image, the pixel differences are more realistic, the matching results are more accurate and reliable, and the corresponding weight should be higher. In this embodiment, considering that the greater the difference in grayscale values ​​of pixels in an image, the clearer the image, and the smaller the difference in grayscale values ​​of pixels (the image becomes blurred and difficult to distinguish), the more blurry the image, the imaging quality of each grayscale image can be determined based on the discreteness of all pixels in the preset fastener pixel area (the more discrete, the greater the difference in grayscale values, and the clearer the image).

[0054] Considering the different emphases in different scenarios—for example, some scenarios prioritize shooting angle while others prioritize image quality—this invention employs a weighted fusion method that combines proximity and image quality to determine the matching weight value for each grayscale image. As an example, in a specific implementation of this invention, for any grayscale image (the i-th grayscale image), its corresponding matching weight value... Represented as: ,in, This indicates the degree of similarity between the i-th grayscale image and the others. This represents the imaging quality of the i-th image; Weights representing the degree of proximity; The weights representing image quality, and and The sum is a positive integer 1. In this embodiment... and All values ​​are set to 0.5 to balance the impact of shooting angle and image quality on the matching results. However, when lighting conditions are good in a real-world scenario, the value can be appropriately increased. (For example, if the value is 0.6, then) This strengthens the weighting of the shooting angle, and when there is significant image shake or interference in reality, the weighting can be increased. (For example, if the value is 0.65, then) This highlights the constraints on image quality.

[0055] A larger matching weight value indicates that the corresponding grayscale image should occupy more weight in subsequent calculations. Therefore, the matching weight can be expressed as: ,in, This represents the matching weight of the i-th grayscale image; This term represents the sum of the matching weight values ​​of all grayscale images; M represents the number of grayscale images. In this embodiment, grayscale images are acquired from 3 perspectives, so M=3; When the denominator is 0, it means that the matching situation between all grayscale images and their corresponding optimal template images is extremely poor. It is considered that the grayscale image capture quality is too poor to be used for subsequent defect detection. At this time, the system does not perform any subsequent calculations.

[0056] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the comprehensive matching value includes: Based on the matching weight and corresponding matching degree of each grayscale image, a reference matching value for each grayscale image is calculated; the reference matching value is positively correlated with the matching weight and the matching degree, respectively. A comprehensive matching value is determined by combining all the aforementioned reference matching values. In this embodiment, the comprehensive matching value is determined based on the sum of all the aforementioned reference matching values.

[0057] A higher matching degree indicates a stronger consistency between the grayscale image and the corresponding optimal template image at the pixel grayscale level, and a closer proximity to a complete and unmissing state of the fastener from that viewpoint (not implying completeness from other viewpoints). A higher matching weight indicates better imaging quality of the grayscale image and a closer proximity of the shooting angle to the optimal template, resulting in higher reliability and reference value of the matching result in comprehensive judgment. Based on the above analysis, both the matching weight and matching degree are positively correlated with the matching situation of the grayscale image. In this embodiment of the invention, the reference matching value can be expressed by the following formula: ,in, This represents the reference matching value of the i-th grayscale image, and its value range is [0,1]. This represents the matching degree of the i-th grayscale image.

[0058] By fusing the reference matching values ​​of grayscale images from all perspectives, a comprehensive matching value is obtained. This comprehensively integrates the matching information from each perspective and avoids the random deviations that may exist from a single perspective. Therefore, in a specific implementation of this invention, the sum of all reference matching values ​​is used as the comprehensive matching value. The larger the value, the closer the grayscale image is to the complete and flawless standard template image, that is, the smaller the probability of missing parts.

[0059] Preferably, in some possible implementations of the embodiments of the present invention, the process of detecting missing railway fasteners includes: If the overall matching value is greater than the preset integrity threshold, the railway fastener is determined to be in an intact state. If the overall matching value is less than or equal to the preset integrity threshold, the railway fastener is determined to be in a missing state.

[0060] In one implementation of the present invention, the preset integrity threshold can be obtained through experimental calibration. Specifically, N (e.g., 20) fastener samples with slight defects (e.g., missing a screw or nut, to increase the upper limit of integrity judgment and ensure the safety of testing in actual scenarios) are selected in advance and fixed on the railway. According to the aforementioned method for calculating the comprehensive matching value, N comprehensive matching values ​​with defects are calculated, and the maximum value of these N comprehensive matching values ​​with defects is taken as the preset integrity threshold. In addition, the preset integrity threshold can also be taken within the empirical range [0.6, 0.8], and in this embodiment, the preferred value is 0.8.

[0061] If the overall matching value is greater than the preset integrity threshold, it means that the fastener is not slightly missing (very complete). Conversely, if the overall matching value is less than or equal to the preset integrity threshold, it means that the fastener is slightly missing, which may develop into a serious missing condition later. Therefore, the railway fastener is directly judged to be in a missing state, which significantly improves the accuracy and reliability of fastener missing detection.

[0062] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the degree of proximity includes: By performing a negative correlation mapping between the Euclidean distance of the center coordinates of each grayscale image and the center coordinates of the optimal template image, the degree of closeness of each grayscale image can be obtained.

[0063] The center coordinates of a grayscale image reflect the shooting angle of the image. The closer the Euclidean distance between the center coordinates of two images, the closer their shooting angles are. Conversely, the farther the Euclidean distance, the greater the difference in their shooting angles. Therefore, in a specific implementation of this invention, for each grayscale image, the Euclidean distance between its center coordinates and the center coordinates of the corresponding optimal template image is calculated and normalized to the maximum and minimum values ​​to obtain the proximity distance of each grayscale image (range [0,1]). Since the larger the proximity distance, the greater the degree of proximity, i.e., the two are negatively correlated, each proximity distance is further negatively correlated. Specifically, each proximity distance is subtracted from the positive integer 1, and the result is taken as the degree of proximity of each grayscale image.

[0064] It should be noted that the maximum value used in the maximum-minimum normalization process can be set to one-tenth of the grayscale image size (rounded to the nearest integer; for example, if the longer side of the grayscale image has 1024 pixels, the maximum value is 102). This indicates that the Euclidean distance between the center coordinates of the grayscale image and the center coordinates of the optimal template image is considered to be no more than one-tenth of the image length. The minimum value used is set to 0, indicating that the center coordinates of the grayscale image coincide with the center coordinates of the optimal template image. This can also be adjusted by the implementer. If the calculated Euclidean distance is greater than this maximum value, the corresponding normalized proximity distance is forcibly set to a positive integer of 1.

[0065] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the imaging quality includes: Each grayscale image is convolved with the Laplacian operator to determine each convolved image; The variance of all pixels in the preset fastener pixel region of each convolutional image is normalized to obtain the imaging quality of each grayscale image.

[0066] In a clear image, the edge grayscale changes drastically with a large second-order rate of change of grayscale values, while in a blurred image, the edge grayscale changes are gradual with a small second-order rate of change of grayscale values. Therefore, in this embodiment, each grayscale image is convolved with a Laplacian operator to obtain a convolved image. The Laplacian operator can be a standard configuration... The basic symmetric Laplacian kernel (OpenCV cv2.Laplacian default kernel): In order to ensure that the grayscale image size remains unchanged before and after convolution, zero padding is applied around the grayscale image before the convolution operation. That is, a ring of pixels with a grayscale value of 0 is added to the outermost edge of the grayscale image. The specific process is a well-known technique to those skilled in the art and will not be described in detail here.

[0067] The larger the variance of all pixels in the preset fastener pixel region of the convolutional image, the clearer the image; that is, the variance is positively correlated with the imaging quality. Therefore, the imaging quality of each grayscale image can be obtained simply by normalizing the variance. As an example, in a specific implementation of this invention, the variance calculated in the preset fastener pixel region of each convolutional image is first obtained. The variances corresponding to all convolutional images are then input into a maximum-minimum normalization algorithm, and the normalized variance is output, which is the imaging quality of each grayscale image. The maximum and minimum values ​​used in this maximum-minimum normalization algorithm are obtained according to the actual situation. For example, when multiple values ​​can be obtained during the implementation process and it is necessary to compare the magnitude relationships between different values, multiple values ​​can be statistically analyzed to obtain the maximum and minimum values. However, when only a single value can be obtained during the implementation process or it is not necessary to compare the magnitude relationships between different values, the maximum and minimum values ​​can be statistically analyzed based on a large amount of historical experimental data or prior data obtained earlier.

Claims

1. A method for detecting missing railway fasteners based on machine vision, characterized in that, The method includes: Acquire grayscale images of the target railway fastener from each viewpoint; the grayscale images include the grayscale value and pixel coordinates of each pixel; acquire a standard template image library, which is based on railway fasteners without missing data from all viewpoints; Based on the positional change trend of the pixel coordinates of all pixels representing the same real-world meaning in the grayscale image, a standard template image library is used to filter and determine the corresponding optimal template image; based on the relative offset between the grayscale values ​​of each pixel in the grayscale image and the corresponding optimal template image, the corresponding matching degree is determined; based on the overall similarity between the grayscale image and the corresponding optimal template image, the corresponding matching weight is analyzed and determined. Based on the matching weight, all the matching degrees are fused to determine a comprehensive matching value; based on the comprehensive matching value, railway fastener missing detection is performed.

2. The method for detecting missing railway fasteners based on machine vision according to claim 1, characterized in that, The method for obtaining the optimal template image includes: Extract the corresponding fastener region from each grayscale image; perform line fitting on all pixels in each grayscale image to construct a rail fitting line and a sleeper fitting line respectively; determine the center coordinates of each grayscale image based on the intersection between the rail fitting line and the sleeper fitting line. In the fastener region of each grayscale image, the template distance between each grayscale image and each standard template image in the standard template image library is obtained based on the spatial distance between the center coordinates of each grayscale image and the center coordinates of each standard template image in the standard template image library. The standard template image corresponding to the minimum value of the template distance is used as the optimal template image for each grayscale image.

3. The method for detecting missing railway fasteners based on machine vision according to claim 2, characterized in that, The method for obtaining the matching degree includes: Based on the relative spatial position between the center coordinates of each grayscale image and the corresponding optimal template image, each grayscale image is aligned to determine the image to be tested; Based on the pixel grayscale difference between each pixel in the preset fastener pixel area of ​​each test image and the corresponding optimal template image in the preset fastener pixel area, the matching reference degree of each pixel in the preset fastener pixel area of ​​each test image is obtained. The matching degree of each grayscale image is calculated based on the overall size of the matching reference degree of all pixels in the preset fastener pixel area of ​​each image to be tested.

4. The method for detecting missing railway fasteners based on machine vision according to claim 3, characterized in that, The method for obtaining the matching weight includes: The degree of similarity between each grayscale image and the center coordinates of the optimal template image is determined based on the relative deviation between the center coordinates of each grayscale image and the center coordinates of the optimal template image. The imaging quality of each grayscale image is determined based on the discreteness of all pixels in the preset fastener pixel area of ​​each grayscale image; The proximity and the imaging quality are weighted and fused to determine the matching weight value for each grayscale image; The matching weight of each grayscale image is determined based on the proportion of each matching weight value among all matching weight values.

5. The method for detecting missing railway fasteners based on machine vision according to claim 1, characterized in that, The method for obtaining the comprehensive matching value includes: Based on the matching weight and corresponding matching degree of each grayscale image, a reference matching value for each grayscale image is calculated; the reference matching value is positively correlated with the matching weight and the matching degree, respectively. By combining all the aforementioned reference matching values, a comprehensive matching value is determined.

6. The method for detecting missing railway fasteners based on machine vision according to claim 1, characterized in that, The process of detecting missing railway fasteners includes: If the overall matching value is greater than the preset integrity threshold, the railway fastener is determined to be in an intact state. If the overall matching value is less than or equal to the preset integrity threshold, the railway fastener is determined to be in a missing state.

7. The method for detecting missing railway fasteners based on machine vision according to claim 3, characterized in that, The difference in grayscale values ​​between each pixel in the preset fastener pixel region of each image under test is calculated using the absolute value of the difference. This difference is compared with the grayscale values ​​of each pixel in the corresponding optimal template image in the preset fastener pixel region.

8. The method for detecting missing railway fasteners based on machine vision according to claim 4, characterized in that, The method for obtaining the degree of proximity includes: By performing a negative correlation mapping between the Euclidean distance of the center coordinates of each grayscale image and the center coordinates of the optimal template image, the degree of closeness of each grayscale image can be obtained.

9. The method for detecting missing railway fasteners based on machine vision according to claim 4, characterized in that, The method for obtaining the imaging quality includes: Each grayscale image is convolved with the Laplacian operator to determine each convolved image; The variance of all pixels in the preset fastener pixel region of each convolutional image is normalized to obtain the imaging quality of each grayscale image.

10. A method for detecting missing railway fasteners based on machine vision according to claim 6, characterized in that, The preset integrity threshold is set to 0.8.

Images