Train defect detection method and system and computer program

Through dual recognition and verification technology, combined with preprocessing, feature matching and affine transformation of RGB graphs and depth graphs, the problems of low efficiency and high false alarm rate of existing train defect detection methods are solved, and higher detection accuracy is achieved and effects.

CN119067927AActive Publication Date: 2024-12-03SHANGHAI SHIKAI INTELLIGENT TECHNOLOGY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411094636.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-10
Publication Date
2024-12-03
Estimated Expiration
2044-08-10

Smart Images

  • Figure CN119067927A_ABST
    Figure CN119067927A_ABST
Patent Text Reader

Abstract

The invention discloses a train defect detection method. The method comprises the following steps: acquiring a template and a to-be-detected RGB image of a train, and acquiring a template and a to-be-detected depth image of the train; preprocessing the RGB image to obtain a template and a grey-scale image to be detected; performing feature matching on the template and the to-be-detected grey-scale map to obtain a homography matrix; performing affine transformation on the to-be-detected grey-scale map and the to-be-detected depth map by using the homography matrix according to the template grey-scale map and the template depth map to obtain the transformed to-be-detected grey-scale map and the transformed to-be-detected depth map; performing flaw detection on the grey-scale map and the depth map to obtain a grey-scale flaw position map and a depth flaw position map; and comparing the gray level flaw position map with the depth flaw position map, and determining the real flaw spot of the train. According to the technical scheme provided by the invention, the influence of factors such as illumination and camera position transformation is eliminated to a certain extent through a technology for performing dual identification and verification on the RGB image and the depth image, the detection accuracy is improved, and a better detection effect is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of defect detection, and in particular to a train defect detection method. Background Art

[0002] There are many types of trains in China, including EMUs, high-speed trains, subways, etc. It is inevitable that problems will gradually appear in the operation of trains. For the safety of passengers, trains need to be regularly inspected. However, inspections require a lot of manpower and material resources, and manual inspections can usually only be performed at night when the trains are stopped, which adds a lot of difficulty to the inspection work.

[0003] The average inspection time by technical personnel is long and the efficiency is low. Moreover, the inspection effect is easily affected by human factors, which may lead to false detection and missed detection.

[0004] There are currently some defect detection methods based on artificial intelligence, but due to problems such as lighting, shooting environment, shooting equipment, and limitations of the algorithm itself, the detection effect is not good, and it is difficult to fully apply it to trains. It is easy to miss or report a false alarm, and a large amount of manpower is still required to review the detection results. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the current technology, the present invention provides a train defect detection method, which improves the detection accuracy and achieves better detection effect through dual identification and verification technology.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] A train defect detection method comprises the following steps:

[0008] Obtain the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected respectively;

[0009] Preprocess the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected;

[0010] Perform feature matching on the template grayscale image and the grayscale image to be detected to obtain the homography matrix;

[0011] Comparing the template grayscale image and the template depth image, using the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected, to obtain the transformed grayscale image to be detected and the depth image to be detected;

[0012] Perform defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map;

[0013] Compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

[0014] According to one aspect of the present invention, the RGB image and the depth image are obtained by photographing with a camera that supports photographing the RGB image and the depth image.

[0015] According to one aspect of the present invention, the camera performs alignment processing when capturing the RGB image and the depth image.

[0016] According to one aspect of the present invention, the preprocessing of the template RGB image and the RGB image to be detected includes the following steps: converting the RGB image into a grayscale image; performing median filtering on the grayscale image; and performing non-restrictive adaptive histogram equalization on the grayscale image.

[0017] According to one aspect of the present invention, the step of performing feature matching on the template grayscale image and the grayscale image to be detected to obtain a homography matrix comprises the following steps:

[0018] Use the ORB feature detector to perform feature detection on the image and obtain feature points;

[0019] Use brute force matcher to match feature points and obtain matching points;

[0020] Cross-validation and RANSAC algorithms are used to remove incorrect matching points;

[0021] According to the matching points, get the homography matrix.

[0022] According to one aspect of the present invention, the distance type used by the brute force matcher is Hamming distance.

[0023] According to one aspect of the present invention, the defect detection is performed on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain the grayscale defect position map and the depth defect position map in the following manner:

[0024] Perform subtraction, binarization, and corrosion processing on the template grayscale image and the transformed grayscale image to be detected to obtain a grayscale defect position map;

[0025] The template depth map and the transformed depth map to be detected are subjected to difference, binarization, and corrosion processing to obtain a depth defect position map.

[0026] According to one aspect of the present invention, the real defect point is: a position where the pixel value is 255 on the grayscale defect position map and the depth defect position map.

[0027] A train defect detection system, based on the train defect detection method as described above, comprises:

[0028] The image acquisition module obtains the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected;

[0029] The image preprocessing module preprocesses the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected;

[0030] The feature matching module performs feature matching on the template grayscale image and the grayscale image to be detected to obtain the homography matrix;

[0031] An affine transformation module is used to compare the template grayscale image and the template depth image, and use the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected to obtain the transformed grayscale image to be detected and the depth image to be detected;

[0032] The defect detection module is used to perform defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map;

[0033] The defect confirmation module is used to compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

[0034] A computer program, which, when executed, implements the steps of the train defect detection method as described above.

[0035] Advantages of the present invention:

[0036] The present invention provides a train defect detection method, which improves the detection accuracy and achieves better detection effect by performing dual recognition and verification technology on RGB images and depth images.

[0037] Dual identification and verification eliminates the influence of a series of environmental factors such as lighting and camera position changes to a certain extent, while also eliminating the limitations caused by the lack of accuracy of depth maps. Therefore, compared with traditional algorithms, the detection effect is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a train defect detection method according to the present invention;

[0040] Figure 2 It is the template RGB image of the train detection location;

[0041] Figure 3 It is the RGB image to be detected at the train detection site;

[0042] Figure 4 It is the corresponding map of matching points at the train detection location;

[0043] Figure 5 Grayscale defect location map of the train inspection location. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] like Figure 1 As shown, a train defect detection method comprises the following steps:

[0047] S1: Obtain the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected respectively.

[0048] The depth map is a grayscale image in which each pixel represents the distance from a point in the scene to the camera.

[0049] The RGB image and the depth image are obtained by photographing with a camera that supports photographing the RGB image and the depth image.

[0050] When the camera captures the RGB image and the depth image, an alignment process is performed.

[0051] When taking train images, a binocular camera that can simultaneously capture RGB images and depth images is used.

[0052] In practical applications, the MV-DB500S Hikvision RGB-D binocular intelligent stereo camera is used to shoot and obtain the template RGB image and the RGB image to be detected. For example: Figure 2 The following is a template RGB image of a certain detection point of a train taken by a camera. Figure 3 Shown is the RGB image to be detected.

[0053] The MV-DB500S Hikvision RGB-D binocular intelligent stereo camera is used to shoot and obtain the depth map of the template image and the image to be detected. The camera has the function of setting the alignment of the RGB image and the depth map. When this setting is turned on during shooting, the RGB image and depth map obtained are aligned.

[0054] In practical applications, due to factors such as changes in lighting or camera position, the template image and the image to be detected cannot be completely aligned or some lighting is misjudged as a defect. Therefore, using a depth map for dual recognition and verification can improve detection accuracy and achieve better detection results.

[0055] S2: Preprocess the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected.

[0056] The preprocessing of the template RGB image and the RGB image to be detected includes the following steps: converting the RGB image into a grayscale image; performing median filtering on the grayscale image; and performing non-restrictive adaptive histogram equalization on the grayscale image.

[0057] To perform operations such as feature matching on images, it is necessary to convert the RGB image into a grayscale image before processing.

[0058] Median filtering is a nonlinear signal processing technology based on sorting statistics theory that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median of the values ​​of all points in a neighborhood of that point, so that the surrounding pixel values ​​are close to the true value, thereby eliminating isolated noise points.

[0059] Unrestricted adaptive histogram equalization improves the contrast of an image by calculating the local histogram of the image and redistributing the brightness. Compared with the traditional histogram equalization method, adaptive histogram equalization is localized and performs equalization based on the local area of ​​the image, thus avoiding the problem of over-enhancement or distortion that may be caused by global equalization.

[0060] The principle of non-restrictive adaptive histogram equalization is to divide the image into many small areas or templates, and then perform histogram equalization on the pixels in each template. This process is adaptive because it adjusts the equalization effect according to the local characteristics of the image. For example, if the image in a certain area is darker, the brightness of this area will be increased to enhance the contrast and visual effect of the image; on the contrary, if the image in a certain area is brighter, the brightness of this area will be reduced to prevent the image from being overexposed or distorted.

[0061] Image preprocessing can enhance image contrast and reduce the impact of image defects on the image.

[0062] S3: Perform feature matching on the template grayscale image and the grayscale image to be detected to obtain a homography matrix.

[0063] The step S3 of performing feature matching on the template grayscale image and the grayscale image to be detected to obtain a homography matrix includes the following steps:

[0064] S31: Use the ORB feature detector to perform feature detection on the image to obtain feature points.

[0065] The ORB (Oriented FAST and Rotated BRIEF) feature detector combines the FAST (Features from Accelerated Segment Test) key point detection and the BRIEF (Binary Robust Independent Elementary Features) descriptor, while adding rotation invariance and scale invariance.

[0066] The main principle of ORB feature detector can be divided into the following three steps:

[0067] (1) Extracting feature points: ORB feature points are boundaries or places where grayscale changes greatly. The definition of ORB feature points is points that are different from the surrounding points.

[0068] (2) Calculate descriptors: Once feature points are detected, the ORB algorithm calculates a descriptor for each feature point. ORB uses the BRIEF descriptor, which generates a binary string by comparing pixel pairs around the feature point. At the same time, ORB improves BRIEF to make it rotation invariant.

[0069] (3) Matching feature points: After obtaining the descriptors of the feature points, find the matching feature points by comparing the descriptors of the feature points in different images.

[0070] S32: Match feature points using a brute force matcher to obtain matching points.

[0071] The distance type used by the brute force matcher is the Hamming distance.

[0072] The brute force matcher based on Hamming distance is mainly used to deal with the matching problem between binary feature descriptors (such as ORB, BRIEF, etc.). The Hamming distance is particularly useful in this scenario because it measures the number of different bits between two binary strings of equal length, which can directly reflect the difference between binary feature descriptors.

[0073] The brute force matcher works as follows: (1) For each feature descriptor in the first image, the Hamming distance between it and all feature descriptors in the second image is calculated; (2) the descriptor pair with the smallest Hamming distance is selected as the match.

[0074] The brute force matcher does not require any training or learning process, so it can be easily used in various application scenarios. In addition, since the brute force matcher directly calculates the similarity between feature descriptors, it is more robust to factors such as illumination changes, rotation and scale changes. Due to the above advantages of the brute force matcher, when it is used for train defect detection, it can eliminate the influence of a series of environmental factors such as illumination and camera position changes on detection to a certain extent.

[0075] S33: Use cross validation and RANSAC algorithm to remove incorrect matching points.

[0076] Cross-validation is used to cross-validate matching points: for each pair of matching points, one of the points is used as the query point, and its best matching point is searched in the other image. If the best matching point found is the same as or very close to the original matching point, the pair of matching points is considered reliable; otherwise, it is considered an incorrect matching point and removed. Exchange the query and search images, transform the query points and search matching points, and perform cross-validation.

[0077] The RANSAC algorithm can be used to remove incorrect matching points. The principle of the RANSAC algorithm is: (1) Randomly select several pairs of matching points as sample points, use these sample points to estimate a transformation model, and then calculate the distance between all matching points and this model; (2) Matching points with a distance less than a certain threshold are considered to be inliers (i.e., correct matching points), while those with a distance greater than the threshold are considered to be outliers (i.e., incorrect matching points); (3) Through multiple iterations, the model with the largest number of inliers is selected as the optimal model, and matching points that are inconsistent with the model are considered to be incorrect matching points and removed.

[0078] For example: Figure 2 The template RGB image and Figure 3 After image preprocessing of the RGB image to be detected, feature matching is performed to obtain matching points. The corresponding results of the matching points on the original RGB image are as follows Figure 4 shown.

[0079] S34: Obtain a homography matrix according to the matching points.

[0080] The homography matrix is ​​a 3x3 matrix that describes the projection transformation from one plane to another, including rotation, translation, scaling, and tilting. For example, the homography matrix is ​​H in formula (1), and the point (x1, y1) on the first image and the point (x2, y2) on the second image have the mapping relationship of formula (2).

[0081]

[0082] When calculating the homography matrix, the least squares method is usually used to solve a system of linear equations. Specifically, assume that there are N pairs of matching points, each pair of matching points consists of a source point coordinate (x_s, y_s) and a target point coordinate (x_t, y_t); construct two equations for each pair of matching points, which describe the linear relationship between the source point coordinates and the target point coordinates; combine these equations into a linear system of equations, and use the least squares method to solve this system of equations to obtain the elements of the homography matrix.

[0083] S4: Compare the template grayscale image and the template depth image, and use the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected to obtain the transformed grayscale image to be detected and the depth image to be detected.

[0084] The specific implementation of step S4 is:

[0085] (1) First, convert the homography matrix into an affine transformation matrix: Since the homography matrix contains the information of perspective transformation, while the affine transformation does not include perspective transformation, the homography matrix needs to be processed to approximate the affine transformation. Set the third row of the homography matrix to

[001] to obtain an approximate affine transformation matrix.

[0086] (2) Perform affine transformation: Perform matrix multiplication operation on the affine transformation matrix and the point coordinates in the source image to map the points in one image to another image. Perform affine transformation on the template grayscale image, the grayscale image to be detected, the template depth image, and the depth image to be detected, respectively, to obtain the transformed template grayscale image, the grayscale image to be detected, the template depth image, and the depth image to be detected.

[0087] S5: performing defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map.

[0088] The defect detection is performed on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain the grayscale defect position map and the depth defect position map in the following manner:

[0089] Perform subtraction, binarization, and corrosion processing on the template grayscale image and the transformed grayscale image to be detected to obtain a grayscale defect position map;

[0090] The template depth map and the transformed depth map to be detected are subjected to difference, binarization, and corrosion processing to obtain a depth defect position map.

[0091] The principle of subtracting two images is to subtract the corresponding pixel values ​​of the two images to generate a new difference image. Subtracting is usually used to compare the difference between two images, especially for images taken at different times or under different conditions.

[0092] The principle of image binarization is to convert the grayscale image into a binary image, that is, each pixel of the image has only two possible values ​​0 and 255 (representing black and white respectively). In the binarization process, a suitable threshold needs to be selected. All pixels with grayscale values ​​greater than or equal to the threshold are set to 255 (white) in the binary image; and pixels with grayscale values ​​less than the threshold are set to 0 (black) in the binary image.

[0093] The principle of image erosion is as follows: (1) For a given image and structural element (a small-sized shape template), place the structural element at a certain pixel position in the image; (2) If all pixels in the structural element match the pixels in the image (usually the pixel values ​​are equal or meet certain conditions), the pixel remains unchanged; otherwise, the pixel is set to the background value (usually 0 or black); (3) Perform the above operation on each pixel in the image in turn until the entire image is traversed. Erosion can eliminate small noise points, small objects, or disconnected objects in an image.

[0094] When performing corrosion processing, the thresholds for corrosion of the grayscale image and the depth image are inconsistent and need to be processed according to the actual situation.

[0095] For example: Figure 2 and 3 After the above steps are performed on the train inspection part image shown in the figure, the grayscale defect position map can be obtained. Figure 5 As shown, Figure 5 The white part in the image is the defect point.

[0096] S6: Compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

[0097] The actual defect point is the position where the pixel value is 255 on the grayscale defect position map and the depth defect position map. The position where the pixel value is 255 is the white part on the map, such as Figure 5 shown.

[0098] The beneficial effect of this embodiment is that by performing dual recognition and verification technology on the RGB image and depth image of the template image and the image to be detected, the detection accuracy can be improved and a better detection effect can be achieved. Dual recognition and verification can eliminate the influence of factors such as lighting and camera position changes and the limitations caused by the lack of accuracy of the depth map to a certain extent. Therefore, compared with traditional algorithms, this method has a greatly improved detection effect.

[0099] Embodiment 2

[0100] A train defect detection system, based on the train defect detection method as described in the first embodiment, comprises:

[0101] The image acquisition module obtains the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected;

[0102] The image preprocessing module preprocesses the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected;

[0103] The feature matching module performs feature matching on the template grayscale image and the grayscale image to be detected to obtain the homography matrix;

[0104] An affine transformation module is used to compare the template grayscale image and the template depth image, and use the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected to obtain the transformed grayscale image to be detected and the depth image to be detected;

[0105] The defect detection module is used to perform defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map;

[0106] The defect confirmation module is used to compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

[0107] Embodiment 3

[0108] A computer program, which, when executed, implements the steps of the train defect detection method as described in Example 1.

[0109] Embodiment 4

[0110] A readable storage medium stores a computer program as described in Example 3, and when the computer program is executed, the steps of the train defect detection method as described in Example 1 are implemented.

[0111] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A train defect detection method, characterized in that: The following steps are involved: Obtain the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected respectively; Preprocess the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected; Perform feature matching on the template grayscale image and the grayscale image to be detected to obtain the homography matrix; Comparing the template grayscale image and the template depth image, using the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected, to obtain the transformed grayscale image to be detected and the depth image to be detected; Perform defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map; Compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

2. The train defect detection method according to claim 1, characterized in that: The RGB image and the depth image are obtained by photographing with a camera that supports photographing the RGB image and the depth image.

3. The train defect detection method according to claim 2, characterized in that: When the camera captures the RGB image and the depth image, an alignment process is performed.

4. The train defect detection method according to claim 1, characterized in that: The preprocessing of the template RGB image and the RGB image to be detected includes the following steps: converting the RGB image into a grayscale image; performing median filtering on the grayscale image; and performing non-restrictive adaptive histogram equalization on the grayscale image.

5. The train defect detection method according to claim 1, characterized in that: The step of performing feature matching on the template grayscale image and the grayscale image to be detected to obtain a homography matrix comprises the following steps: Use the ORB feature detector to perform feature detection on the image and obtain feature points; Use brute force matcher to match feature points and obtain matching points; Cross-validation and RANSAC algorithms are used to remove incorrect matching points; According to the matching points, get the homography matrix.

6. The train defect detection method according to claim 5, characterized in that: The distance type used by the brute force matcher is the Hamming distance.

7. The train defect detection method according to claim 1, characterized in that: The defect detection is performed on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain the grayscale defect position map and the depth defect position map in the following manner: Perform subtraction, binarization, and corrosion processing on the template grayscale image and the transformed grayscale image to be detected to obtain a grayscale defect position map; The template depth map and the transformed depth map to be detected are subjected to difference, binarization, and corrosion processing to obtain a depth defect position map.

8. The train defect detection method according to claim 1, characterized in that: The real defect point is a position where the pixel value is 255 on the grayscale defect position map and the depth defect position map.

9. A train defect detection system, characterized in that: The train defect detection method according to any one of claims 1 to 8 comprises: The image acquisition module obtains the train's template RGB image, the RGB image to be detected, the template depth image, and the depth image to be detected; The image preprocessing module preprocesses the template RGB image and the RGB image to be detected to obtain the template grayscale image and the grayscale image to be detected; The feature matching module performs feature matching on the template grayscale image and the grayscale image to be detected to obtain the homography matrix; An affine transformation module is used to compare the template grayscale image and the template depth image, and use the homography matrix to perform affine transformation on the grayscale image to be detected and the depth image to be detected to obtain the transformed grayscale image to be detected and the depth image to be detected; The defect detection module is used to perform defect detection on the template grayscale image, the template depth image, the transformed grayscale image to be detected, and the depth image to be detected to obtain a grayscale defect position map and a depth defect position map; The defect confirmation module is used to compare the grayscale defect location map and the depth defect location map to determine the actual defect points of the train.

10. A computer program, characterized in that When the computer program is executed, the steps of the train defect detection method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Pipe defect detection method and device based on gray level image and depth data fusion

    CN114677331A

  • Industrial defect detection method based on few defect samples

    CN114897865A

  • Defect detection method, device, equipment, system and storage medium

    CN118037619A

  • Inspection apparatus and method based on deep learning

    KR102221317B1