Train box grating foreign matter detection method based on 3D machine vision analysis

Abstract

The invention provides a train box grid foreign matter detection method based on 3D machine vision analysis. The train box grid foreign matter detection method comprises the following steps that S1, a marking frame, a datum point cloud picture and a datum envelope box of a to-be-detected position of a grid are obtained through a 3D vision collection device; s2, foreign matter detection: S21, in a foreign matter detection process, a 3D visual acquisition device acquires a three-dimensional point cloud image of a to-be-detected grid, and registration is performed through the reference point cloud picture in the step S1; s22, obtaining a point cloud image to be detected in the registered three-dimensional point cloud image by using a labeling box; obtaining a point cloud of a non-grid area and a point cloud of a grid area in the registered three-dimensional point cloud image by using the reference envelope box; and carrying out Euclidean distance-based KNN clustering on the point clouds in the non-grid region, and if the point clouds can be clustered into at least one type, determining that foreign matters exist. The system can be flexibly applied to various inspection scenes, is less limited by sites, is less influenced by illumination and background, and is more stable.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D machine vision analysis, and in particular to a train box grille foreign body detection method based on 3D machine vision analysis. Background Art

[0002] In the operation and maintenance of modern trains, the train body grille is an important structural component responsible for protecting the inside and outside of the train from external environmental factors (such as flying stones, garbage, etc.). However, as the train runs for a long time, some foreign matter (such as debris, stones, garbage, etc.) may accumulate on the grille surface, which not only affects the appearance of the train, but also may threaten the normal operation and safety of the train.

[0003] Traditional methods for detecting foreign objects in train grilles mostly rely on manual inspections, which is not only labor-intensive and inefficient, but also prone to omissions due to the limitations of manual inspections, making it difficult to ensure the comprehensiveness and accuracy of the inspections. In addition, with the increase in train speeds and the increase in the amount of inspection tasks, the safety hazards and efficiency issues of manual inspections have become more prominent. Therefore, there is an urgent need for an automated, high-precision, and fast foreign object detection method.

[0004] At present, foreign body detection technology based on image processing has made some progress. However, traditional two-dimensional (2D) image processing technology is affected by factors such as changes in illumination and viewing angle, resulting in unstable detection results. Especially in complex environments, 2D image processing methods often cannot effectively distinguish between targets and backgrounds, and have poor recognition accuracy and robustness, which cannot meet the needs of train grille foreign body detection.

[0005] With the development of three-dimensional vision (3D vision) technology, foreign body detection methods based on three-dimensional point cloud data have gradually become an effective solution. Three-dimensional point cloud data can provide richer spatial information, overcoming the shortcomings of two-dimensional images in depth perception and environmental changes. However, foreign body detection based on three-dimensional point cloud data still faces many challenges, including point cloud registration, noise processing, identification and extraction of foreign body areas, etc. In the prior art, point cloud registration algorithms (such as ICP algorithms) are widely used for point cloud alignment from different perspectives, but how to accurately and quickly realize the processing of large-scale point cloud data and accurate identification of foreign objects is still a difficulty in three-dimensional vision applications. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and provide a train box grille foreign body detection method based on 3D machine vision analysis that is less affected by light and background and has strong adaptability.

[0007] The technical solution provided by the present invention is a method for detecting foreign matter in a train box grille based on 3D machine vision analysis, comprising the following steps:

[0008] S1, obtaining a labeling box, a reference point cloud map and a reference envelope box of a position to be detected of the grid through a 3D vision acquisition device;

[0009] S2. Detect foreign matter

[0010] S21, during the foreign body detection process, the 3D visual acquisition device acquires a three-dimensional point cloud image of the grid to be detected, and aligns it with the reference point cloud image described in step S1;

[0011] S22, using the annotation frame to obtain a point cloud image to be detected in the registered three-dimensional point cloud image;

[0012] Using the reference envelope box, a point cloud of a non-grid area and a point cloud of a grid area in the registered three-dimensional point cloud image are obtained;

[0013] KNN clustering based on Euclidean distance is used for point clouds in non-grid areas. If they can be clustered into at least one category, it is considered that there is a foreign object.

[0014] The technical solution of the present invention is further defined as follows: the specific steps of obtaining the reference point cloud image and the reference envelope box by the 3D vision acquisition device in step S1 are:

[0015] S11, a 3D vision acquisition device acquires a visible light image and a depth image of the grid to be detected;

[0016] S12, marking the position of the grid to be detected in the visible light image, and obtaining a marking frame {x 1 ,y 1 ,x 2 ,y 2}, where x 1 ,y 1 are the X-axis and Y-axis coordinates of the upper left corner of the annotation box, respectively. 2 ,y 2 are respectively the X-axis and Y-axis coordinates of the lower right corner of the marking box;

[0017] S13, according to the annotation box obtained in step S2, extracting the depth of the corresponding position of the envelope box in the depth map, and converting it into a point cloud map as a reference point cloud map;

[0018] S14, taking the envelope box of the reference point cloud as the reference envelope box {x 1 ,y 1 ,z 1 ,x 2 ,y 2 ,z 2}, where x1 ,y 1 ,z 1 are the minimum values ​​of the X-axis, Y-axis, and Z-axis in the reference point cloud, respectively. 2 ,y 2 ,z 2 are the maximum values ​​of the X-axis, Y-axis and Z-axis in the reference point cloud image respectively.

[0019] Furthermore, the steps of aligning the reference point cloud images in step S21 are as follows: first, downsample the two point cloud images, and then use the ICP alignment algorithm to generate a rotation displacement matrix using the difference between the centroid and the eigenvector of the two point sets as the residual, and iterate repeatedly until the exit condition is met to complete the point cloud alignment.

[0020] Furthermore, the specific operation of the registration is: rotate the point cloud selected in step 1 to the point cloud in step 2, set the rotation matrix to P, and use the inverse matrix of P to register the point cloud obtained by the current inspection to the point cloud in step 1.

[0021] Furthermore, in step S22, the specific steps of obtaining the point cloud of the non-grid area and the point cloud of the grid area in the registered three-dimensional point cloud image are as follows: the annotation box is to confirm the area to be detected through the two-dimensional plane, that is, the grid area; the reference envelope box is the envelope box of the grid area in the three-dimensional space during calibration;

[0022] If there are foreign objects, when detecting foreign objects, the point cloud obtained using the annotation box will inevitably have areas beyond the envelope box, then the grid area point cloud and the non-grid area point cloud can be obtained. The non-grid area point cloud needs to be subsequently judged whether it is a foreign object, and the grid area point cloud does not need subsequent processing.

[0023] Beneficial effects: The present invention proposes a method for detecting foreign objects in train box grilles based on 3D machine vision analysis. It utilizes the spatial characteristics of three-dimensional point cloud data, aligns multiple point cloud images through a registration algorithm, and uses a clustering algorithm to effectively identify and locate foreign objects, thereby achieving automated and accurate detection of train box grilles. It can be flexibly used in a variety of inspection scenarios with little site restrictions. At the same time, because three-dimensional point cloud images are used, they are less affected by light and background and will be more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a train box grille foreign body detection method based on 3D machine vision analysis provided by the present invention;

[0025] Figure 2 is a grid point cloud with foreign matter in the embodiment;

[0026] Figure 3 is a schematic diagram of the marked box in the embodiment (red box);

[0027] Figure 4 is the non-grid area point cloud (foreign matter) extracted in the embodiment;

[0028] Figure 5 is the grid area point cloud extracted in the embodiment. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0030] Example 1

[0031] Foreign matter often appears on the grille surface of the rail train. When foreign matter appears on the grille, the point cloud will show a huge difference (see Figure 2 ), one difference comes from the occlusion of foreign objects, and the other difference comes from the protruding point cloud produced by foreign objects. By comparing with the calibration point cloud, it is easy to distinguish whether there is occlusion by foreign objects.

[0032] This embodiment provides a method for detecting foreign objects in a train box grille based on 3D machine vision analysis, which is specifically performed in the following steps.

[0033] S1. Obtaining a labeling box, a reference point cloud map and a reference envelope box of a position to be detected of the grid through a 3D vision acquisition device.

[0034] The specific steps of obtaining the reference point cloud image and the reference envelope box by the 3D vision acquisition device in step S1 are:

[0035] S11. The robot moves to the grid inspection point, faces the grid, and uses a 3D vision acquisition device to take a visible light image and a depth image of the grid to be inspected.

[0036] S12, marking the position of the grid to be detected in the visible light image, such as Figure 3 As shown, the label box is {x 1 ,y 1 ,x 2 ,y 2}, where x 1 ,y 1 are the X-axis and Y-axis coordinates of the upper left corner of the annotation box, respectively. 2 ,y 2 are respectively the X-axis and Y-axis coordinates of the lower right corner of the annotation box.

[0037] S13. According to the annotation box obtained in step S2, the depth of the position corresponding to the envelope box in the depth map is extracted, and converted into a point cloud map as a reference point cloud map.

[0038] S14, taking the envelope box of the reference point cloud as the reference envelope box {x1 ,y 1 ,z 1 ,x 2 ,y 2 ,z 2}, where x 1 ,y 1 ,z 1 are the minimum values ​​of the X-axis, Y-axis, and Z-axis in the reference point cloud, respectively. 2 ,y 2 ,z 2 are the maximum values ​​of the X-axis, Y-axis and Z-axis in the reference point cloud image respectively.

[0039] S2. Detect foreign matter

[0040] S21, during the foreign body detection process, the robot moves to the grid inspection point, faces the grid, and uses the 3D visual acquisition device to capture a 3D point cloud image to obtain a 3D point cloud image of the grid to be detected. Due to the instability of the robot, it is impossible to ensure that the two point clouds are in the same perspective, so they need to be aligned with the reference point cloud map described in step S1.

[0041] The steps for registration through the reference point cloud map are as follows: considering the number of point clouds, first downsample the two point cloud maps, and then use the ICP (Iterative Closet Point) registration algorithm, using the difference between the centroid and the eigenvector of the two point sets as the residual to generate a rotation displacement matrix, and iterate repeatedly until the exit condition is met to complete the point cloud registration.

[0042] The specific operation of the registration is as follows: Considering stability, the point cloud selected in step 1 is rotated to the point cloud in step 2. Let the rotation matrix be P, and the point cloud obtained by the current inspection is registered to the point cloud in step 1 using the inverse matrix of P.

[0043] The device used to collect the 3D point cloud image of the grid to be detected is Zivid, Industrial 3D cameras for random bin picking, logistics, machine tending – Zivid.

[0044] S22. Use the annotation frame to obtain a point cloud image to be detected in the registered three-dimensional point cloud image.

[0045] Using the reference envelope box, the point cloud of the non-grid area in the registered 3D point cloud image is obtained ( Figure 3 ) and the point cloud of the grid area ( Figure 4 ). KNN clustering based on Euclidean distance is used for point clouds in non-grid areas. If they can be clustered into at least one category, it is considered that there is a foreign object.

[0046] Here we explain a concept that the three-dimensional point cloud and the two-dimensional depth map can be changed back and forth through the formula. This step is to put the frame set at the beginning in the two-dimensional depth map and convert the part in the two-dimensional frame into a point cloud.

[0047] The specific steps of obtaining the point cloud of the non-grid area and the point cloud of the grid area in the registered three-dimensional point cloud image are as follows: the annotation box is to confirm the area to be detected through the two-dimensional plane, that is, the grid area; the reference envelope box is the envelope box of the grid area in the three-dimensional space during calibration;

[0048] If there are foreign objects, when detecting foreign objects, the point cloud obtained using the annotation box will inevitably have areas beyond the envelope box, then the grid area point cloud and the non-grid area point cloud can be obtained. The non-grid area point cloud needs to be subsequently judged whether it is a foreign object, and the grid area point cloud does not need subsequent processing.

[0049] This embodiment is mainly aimed at the use of 3D point cloud technology in scenes with large changes, and the accuracy requirement is not high. Here, it is aimed at relatively small scenes with high accuracy requirements. It can be seen from the attached pictures that the shooting equipment is different, and the point cloud imaging quality is very high. Similar applications also include tire detection. The main difference from this patent is that in normal high-precision scenes, the camera is generally required to be stationary. The camera in this scene is based on an inspection robot, and there are differences in each shooting. The above alignment step is the correction process for the shooting differences, and it also uses the characteristic of the grid being a plane, so the scene can be processed quickly.

[0050] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A train box grille foreign body detection method based on 3D machine vision analysis, characterized in that: The steps include: S1, obtaining a labeling box, a reference point cloud map and a reference envelope box of a position to be detected of the grid through a 3D vision acquisition device; S2. Detection of foreign matter: S21, during the foreign body detection process, the 3D visual acquisition device acquires a three-dimensional point cloud image of the grid to be detected, and aligns it with the reference point cloud image described in step S1; S22, using the annotation frame to obtain a point cloud image to be detected in the registered three-dimensional point cloud image; Using the reference envelope box, a point cloud of a non-grid area and a point cloud of a grid area in the registered three-dimensional point cloud image are obtained; KNN clustering based on Euclidean distance is used for point clouds in non-grid areas. If they can be clustered into at least one category, it is considered that there is a foreign object.

2. According to claim 1, a train box grille foreign body detection method based on 3D machine vision analysis is characterized in that: The specific steps of obtaining the reference point cloud image and the reference envelope box by the 3D vision acquisition device in step S1 are: S11, a 3D vision acquisition device acquires a visible light image and a depth image of the grid to be detected; S12, marking the position of the grid to be detected in the visible light image, and obtaining a marking box {x1, y1, x2, y2}, wherein x1 and y1 are respectively the X-axis and Y-axis coordinates of the upper left corner of the marking box, and x2 and y2 are respectively the X-axis and Y-axis coordinates of the lower right corner of the marking box; S13, according to the annotation box obtained in step S2, extracting the depth of the corresponding position of the envelope box in the depth map, and converting it into a point cloud map as a reference point cloud map; S14. Use the envelope box of the reference point cloud as the reference envelope box {x1, y1, z1, x2, y2, z2}, wherein x1, y1, z1 are the minimum values ​​of the X-axis, Y-axis, and Z-axis in the reference point cloud, respectively; and x2, y2, z2 are the maximum values ​​of the X-axis, Y-axis, and Z-axis in the reference point cloud, respectively.

3. The method for detecting foreign matter in a train box grille based on 3D machine vision analysis according to claim 1 is characterized in that: The steps for aligning the reference point cloud in step S21 are as follows: first, downsample the two point cloud images, then use the ICP alignment algorithm, use the difference between the centroid and the eigenvector of the two point sets as the residual to generate a rotation displacement matrix, and iterate repeatedly until the exit condition is met to complete the point cloud alignment.

4. The method for detecting foreign matter in a train box grille based on 3D machine vision analysis according to claim 3 is characterized in that: The specific operation of the registration is: rotate the point cloud selected in step 1 to the point cloud in step 2, set the rotation matrix to P, and use the inverse matrix of P to register the point cloud obtained in the current inspection to the point cloud in step 1.

5. The method for detecting foreign matter in a train box grille based on 3D machine vision analysis according to claim 1 is characterized in that: The specific steps of obtaining the point cloud of the non-grid area and the point cloud of the grid area in the registered three-dimensional point cloud image in step S22 are as follows: the annotation box is to confirm the area to be detected through the two-dimensional plane, that is, the grid area; the reference envelope box is the envelope box of the grid area in the three-dimensional space during calibration; If there are foreign objects, when detecting foreign objects, the point cloud obtained using the annotation box will inevitably have areas beyond the envelope box, then the grid area point cloud and the non-grid area point cloud can be obtained. The non-grid area point cloud needs to be subsequently judged whether it is a foreign object, and the grid area point cloud does not need subsequent processing.

Images