A robot inspection component defect detection method and application thereof

Abstract

The present application belongs to the field of component defect detection, and particularly relates to a robot inspection component defect detection method and application thereof, comprising: converting a component recognition result corresponding to each inspection image obtained by a target recognition network into a two-dimensional test result semantic point cloud; converting all the annotation boxes in a standard image corresponding to the inspection image into a two-dimensional standard semantic point cloud; determining a similarity transformation of the two point clouds to perform point cloud registration thereon; identifying isolated points in the two-dimensional standard semantic point cloud based on the registration result; taking the annotation box information corresponding to the isolated points as the type and position of the missing component; and further performing cropping on each bounding box in the component recognition result of each inspection image and inputting the component abnormal fine category classification network to obtain the component abnormal fine category result corresponding to each bounding box; wherein a random cropping method is used to construct training samples for training the classification network. The present application realizes efficient and high-precision defect detection suitable for actual application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of component defect detection, and more specifically, relates to a method for detecting component defects by a robot and its application. Background Technology

[0002] With the rapid development of rail transit technology in recent years, the total mileage and number of carriages of my country's high-speed rail have been steadily increasing, highlighting the growing pressure on the safety of high-speed trains. Currently, manual maintenance of high-speed trains typically involves using specialized detection equipment to visually inspect pre-set inspection points to determine if there are defects in the train's components. However, this method is labor-intensive and time-consuming. Furthermore, since maintenance is usually carried out at night, personnel are prone to fatigue, severely impacting the speed and accuracy of high-speed train equipment maintenance. Therefore, the commonly used approach is to introduce automated maintenance systems for intelligent maintenance, such as inspection robots or mobile high-speed train fault detection systems.

[0003] Inspection robots are more flexible and have greater development prospects than TEDS (Transportation Tunneling System), but most current research on defect detection technology for high-speed trains is based on TEDS images, which cannot be directly applied to inspection images acquired by robots. Therefore, there is an urgent need for a defect detection method suitable for practical applications to solve the problem of high-precision defect detection for inspection robots. Summary of the Invention

[0004] In response to the shortcomings and improvement needs of existing technologies, this invention provides a method for detecting defects in robot inspection components and its application, aiming to achieve a high-efficiency and high-precision defect detection method suitable for practical application scenarios.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for detecting defects in robot-inspected components is provided, comprising:

[0006] The component recognition results obtained from each inspection image through the target recognition network are converted into a two-dimensional test result semantic point cloud. Based on the acquisition position of each inspection image, a standard image is pre-acquired using position constraint mapping, and all bounding boxes in the standard image are converted into a two-dimensional standard semantic point cloud. The similarity transformation of the two point clouds is determined to perform point cloud registration. For the registered point cloud, the distance between each point in the two-dimensional standard semantic point cloud and its corresponding point in the two-dimensional test result semantic point cloud is calculated. If the distance is greater than a threshold, it is regarded as an isolated point. The bounding box information corresponding to the isolated point is used as the type and location of the missing component to complete the component loss defect diagnosis.

[0007] And / or,

[0008] The bounding boxes in the component recognition results of each inspection image are cropped and input into the component anomaly fine category classification network to obtain the component anomaly fine category results corresponding to each bounding box in the inspection image; wherein, the training samples used to train the classification network include samples obtained in the following way: based on the labeled boxes of each type of component in the training inspection image, several regions are randomly cropped on the image to obtain fine category training samples of that type of component, and the fine category includes normal category, abnormal category and non-component background category.

[0009] Furthermore, the component identification result is obtained in the following way:

[0010] Inspection images of various areas on complex machinery within the workshop are collected. Based on the acquisition position of each inspection image, a standard image is pre-acquired using position constraint mapping. A multi-channel prior semantic image of the corresponding area is then obtained from this standard image. The construction of the multi-channel prior semantic image involves: converting the component annotation information of the standard image to obtain an information form that can express the prior probability of the components on the image; initializing a multi-channel grayscale image with the same number of channels as the total number of all component types in the complex machinery; and superimposing the information form onto the multi-channel grayscale image to generate a multi-channel prior semantic image, serving as tensor-quantified prior structural knowledge. Each channel's prior semantic image represents the prior position and size of a certain component type in the standard image, with pixel values ​​representing probabilities.

[0011] A pre-constructed multi-channel semantic image input branch is used to fuse the last two layers of high semantic feature images generated by the trained inspection image feature extraction network for each inspection image with the corresponding multi-channel prior semantic image, thereby enhancing the inspection image features with prior structural knowledge; wherein, the multi-channel semantic image input branch is constructed by combining the Bottleneck module of ResNet and deformable convolution operators;

[0012] Based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network, the component target recognition result is obtained through regression and classification.

[0013] Furthermore, the similarity transformation is calculated using an improved iterative nearest-point method, specifically as follows:

[0014] An initial similarity transformation is applied to the two-dimensional standard semantic point cloud. For each point in the transformed two-dimensional standard semantic point cloud, nearest neighbor matching with category constraints is performed to find the corresponding point in the two-dimensional test result semantic point cloud, resulting in multiple sets of matching points. A new similarity transformation is solved using the multiple sets of matching points. Based on the new similarity transformation, the above process is iterated until the iteration termination condition is reached, resulting in the first similarity transformation.

[0015] The initial similarity transformation is used to transform the two-dimensional standard semantic point cloud. For each point in the transformed two-dimensional standard semantic point cloud, nearest neighbor matching with category constraints is performed to find the corresponding point in the two-dimensional test result semantic point cloud, resulting in multiple sets of matching points. The matching relationship with the largest distance in the many-to-one matching relationship is removed from the multiple sets of matching points. Then, based on the multiple sets of matching points after the removal operation, a new similarity transformation is solved. Based on the new similarity transformation, the above process is iterated until the iteration termination condition is reached to obtain the second similarity transformation.

[0016] Substituting the first and second similarity transformations into the minimum registration residual f, respectively, yields the corresponding minimum registration residual values. The similarity transformation corresponding to the smaller minimum registration residual value is taken as the final similarity transformation. The minimum registration residual f is expressed as:

[0017] [x ti y ti 1] T =A·[x i y i 1] T ;

[0018]

[0019]

[0020] In the formula, A represents the similarity transformation, and the first and second similarity transformations are substituted into A respectively; x i ,y i This represents the two-dimensional coordinates of a point in a two-dimensional standard semantic point cloud, (x... di ,y di ) represents point (x) i ,y i ) nearest neighbor matching point.

[0021] Furthermore, when the component abnormal fine category classification network outputs fine category results for normal bolts and / or loose bolts, a 3D point cloud measurement algorithm is used to further confirm whether the bolts are indeed loose, specifically:

[0022] The fine-grained classification results of normal bolts and loose bolts are mapped to the bounding boxes corresponding to the component recognition results in the 3D point cloud to obtain the point cloud information corresponding to the bolt area; wherein, the 3D point cloud is the 3D point cloud information generated when each inspection image is acquired;

[0023] The Ransac algorithm is used to segment the first plane from the point cloud information corresponding to the bolt region, and the normal vector of the first plane is calculated. From the points in the point cloud information corresponding to the bolt region that exclude the first plane, the Ransac method with the normal vector of the first plane is used to segment the second plane. The first plane and the second plane are set as the top and bottom planes of the bolt.

[0024] After obtaining the first and second planes, the planar distance is calculated using an approximation method. Specifically, the distance from all points in the second plane to the first plane is calculated. After sorting the distances, the average of the set distance intervals is taken as the distance from the second plane to the first plane, which is the height of the bolt.

[0025] Furthermore, the Ransac method with the first plane normal vector constraint is specifically as follows:

[0026] S1. Randomly select three points from the point cloud information corresponding to the bolt region after excluding the points of the first plane to determine a plane, and calculate the normal vector of the random plane;

[0027] S2. Calculate the cosine value of the angle between the normal vector of the first plane and the normal vector of the random plane using the inner product. If the cosine value of the angle is greater than the preset threshold, the constraint condition is met, indicating that the planes with the two normal vectors are approximately parallel. All points whose distance from the random plane is less than the threshold are classified as points in the plane. If the constraint is not met, the random plane is discarded directly, and the process returns to step S1 until the preset number of iterations is reached.

[0028] S3. Take the random plane with the most points in the plane as the largest plane obtained by partitioning, that is, the second plane.

[0029] Furthermore, the training samples used to train the classification network include samples constructed in the following manner:

[0030] Each bounding box in each training inspection image is used as a cropping box to directly obtain the positive sample with a fine class confidence of 1.0 corresponding to that bounding box;

[0031] The maximum and minimum areas, as well as the maximum and minimum aspect ratios, of each component on the inspection image are statistically analyzed. Several cropping boxes are randomly set for each fine category of each component on the inspection image. First, the center point of the cropping box is obtained by sampling in a uniform distribution on the image area. Then, the area of ​​the cropping box is obtained by sampling in a uniform distribution between the statistically analyzed maximum and minimum areas. Finally, the aspect ratio of the cropping box is obtained by sampling in a uniform distribution between the statistically analyzed maximum and minimum aspect ratios. The specific length and width values ​​are calculated by using the aspect ratio and area to obtain the random cropping box.

[0032] If each cropping box and a corresponding fine-class bounding box in the inspection image have overlapping areas, the Intersection over Union (IoU) ratio between the bounding box and the cropping box is calculated. If each cropping box and multiple bounding boxes of the corresponding fine-class bounding box in the inspection image have overlapping areas, the maximum value of the IoU is taken after calculation. If each cropping box and all bounding boxes of the corresponding fine-class bounding box in the inspection image have no overlapping areas, the IoU is set to 0. If the IoU is greater than or equal to a preset value, the cropping box region is used as a positive sample with a confidence level of 1.0 for the corresponding fine-class; otherwise, the cropping box region is used as a negative sample with a confidence level of (1-IoU).

[0033] The number of positive and negative samples is counted, and additional sampling is used to balance the samples, thus completing the generation of sample labels.

[0034] Furthermore, the inspection image is an inspection image of a rail transit vehicle; the method also includes detecting the thickness of the brake pads on the rail transit vehicle to determine whether the brake pads exceed the limit, specifically:

[0035] Step 1: Extract the image region of the brake pads in the current inspection image and initialize the dual thresholds to (250, 180);

[0036] Step 2: Perform Canny edge detection using the current dual thresholds to obtain the edge response;

[0037] Step 3: Record the coordinates of the edge points on the vertical central axis of the image region, that is, at the midpoint of the X-axis of the edge response, record the coordinates of the edge points from top to bottom in the Y-axis direction;

[0038] Step 4: If there are at least two edge points on the vertical centerline, proceed to Step 5 to calculate the thickness; otherwise, lower both threshold values ​​and return to Step 2.

[0039] Step 5: Using the vertical centerline edge points with the smallest and largest Y values ​​as the upper and lower edge points of the gate plate, respectively, map the coordinates of the upper and lower edge points to the 3D point cloud generated when the camera collects the inspection image to obtain the Y value in the real world coordinate system. Use the difference between the Y values ​​of the upper and lower edge points as the final thickness measurement result.

[0040] Furthermore, the inspection image is an inspection image of a rail transit vehicle; the method also includes detecting the height of the sweeper on the rail transit vehicle to determine whether the sweeper exceeds the limit, specifically:

[0041] Preprocessing is performed on inspection images containing stone sweepers to obtain balanced binary images;

[0042] On the smoothed binary image, based on the bounding box of the sweeper predicted by the object recognition network, the boundary line between the sweeper and the track plane is found. The process is as follows:

[0043] Step S1: Map the bounding box of the sweeper onto the binary image; on the binary image, find the first black pixel vertically downward from the center point of the sweeper bounding box as the lower boundary of the sweeper, and then continue downward to find the first white pixel as the upper boundary of the track plane. Add the coordinate information of these two boundary points and their height difference H to the height difference set.

[0044] S2. Move left and right from the center point of the sweeper bounding box in the X-axis direction, moving 2 coordinates each time, and then repeat step one. In this case, set the coordinate range for the left and right offsets. The final height difference set contains multiple sets of boundary point coordinate information and height difference H.

[0045] S3. Sort the height difference set according to the height difference H, and remove the two groups with the smallest and largest H to obtain a new height difference set;

[0046] S4. Remove outliers by using the mean and median of the height differences H in the new height difference set to obtain the final height difference set;

[0047] S5. Take the smaller X quantile index X_index after sorting by height difference H in the final height difference set, and use the boundary points corresponding to the indices within the range of X_index±A to fit the straight line of the lower boundary of the sweeper and the straight line of the upper boundary of the track plane respectively. Take the average of the slopes of the two lines to obtain a uniform fitting slope k, where X and A are positive numbers.

[0048] S6. The upper boundary point corresponding to X_index is (x0, y... top The lower boundary point is (x0, y0). bottom Passing through these two points respectively, a final upper and lower boundary fitted straight line with a consistent slope k is constructed, with the intercept being:

[0049]

[0050] S7. Based on the intercept, the distance between the fitted straight lines of the final upper and lower boundaries is obtained, which is used as the final height of the sweeper to the track surface.

[0051] Furthermore, the outlier removal method in step S4 is as follows: calculate the mean (Mean) and median (Median) of H. If the height difference H of a certain group of boundary points meets the following condition... or Then the boundary points in this group are considered outliers and are removed from the new set of height differences;

[0052] And / or,

[0053] In step S5, X is four and A is five.

[0054] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed by a processor, it controls the device containing the storage medium to perform a robot inspection component defect detection method as described above.

[0055] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0056] (1) In the first stage of this invention, the type of parts is detected and the parts are located through a trained deep learning network. At the same time, diagnostic algorithms for the two major categories of part defects, namely missing and abnormal, are added after the target recognition network. This can effectively utilize the part recognition results to perform targeted defect diagnosis on each part, which greatly improves the efficiency of defect diagnosis.

[0057] (2) To address component loss defects, this invention uses an iterative nearest-point method combining category constraints and scale transformation to register the two-dimensional semantic point clouds of the standard and test images, obtaining isolated points in the standard image point cloud, thereby diagnosing component loss. Compared to other component loss defect detection methods, this method effectively solves the problems of mismatch in component registration and errors in the position of captured images by incorporating category constraints and scale transformation.

[0058] (3) In the absence of training samples, the present invention designs a method for generating training samples for classification networks based on random pruning, which broadens the distribution of classification samples and can effectively improve the training effect of classification networks and improve the accuracy of classification network prediction.

[0059] (4) After the classification network, a point cloud measurement method is introduced for the two categories of normal bolts and loose bolts. The bolt height is calculated by dividing the 3D point cloud plane using the Ransac sampling method twice, which effectively improves the detection accuracy of loose bolts.

[0060] (5) Regarding the problem of component over-limit defects, this invention introduces two methods, edge detection and line fitting, based on the characteristics of the environment and the component itself, to measure the key values ​​of the component, which can effectively improve the accuracy of the measurement of the key values ​​of the component. Attached Figure Description

[0061] Figure 1 A flowchart of a robot inspection component defect detection method provided in an embodiment of the present invention;

[0062] Figure 2 This is a flowchart of a component loss defect detection algorithm provided in an embodiment of the present invention;

[0063] Figure 3 A flowchart of an improved iterative nearest point algorithm provided in an embodiment of the present invention;

[0064] Figure 4 This is a flowchart of a measurement method for bolt loosening defects provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0066] Example 1

[0067] A method for detecting defects in components inspected by a robot, such as Figure 1 As shown, it includes:

[0068] The component recognition results obtained from each inspection image through the target recognition network are converted into a two-dimensional test result semantic point cloud. Based on the acquisition position of each inspection image, a standard image is pre-acquired using position constraint mapping, and all bounding boxes in the standard image are converted into a two-dimensional standard semantic point cloud. The similarity transformation of the two point clouds is determined to perform point cloud registration. For the registered point cloud, the distance between each point in the two-dimensional standard semantic point cloud and its corresponding point in the two-dimensional test result semantic point cloud is calculated. If the distance is greater than a threshold, it is regarded as an isolated point. The bounding box information corresponding to the isolated point is used as the type and location of the missing component to complete the component loss defect diagnosis.

[0069] And / or,

[0070] The bounding boxes in the component recognition results of each inspection image are cropped and input into the component anomaly fine category classification network to obtain the component anomaly fine category results corresponding to each bounding box in the inspection image; wherein, the training samples used to train the classification network include samples obtained in the following way: based on the labeled boxes of each type of component in the training inspection image, several regions are randomly cropped on the image to obtain fine category training samples of that type of component, and the fine category includes normal category, abnormal category and non-component background category.

[0071] It should be noted that since the manufacturing and installation of components of the same complex machinery (such as high-speed trains) adhere to strict specifications, the spatial structure of components is consistent in images taken at the same acquisition points for the same train model. Both the inspection images and the standard images are acquired by the inspection robot at several specific locations. Therefore, a unique standard image can be mapped based on the acquisition location of the inspection image according to positional constraints. Both the standard image and the training inspection image can be acquired by the robot at fixed points using a 3D camera at the end effector of the robotic arm. The mapping can be implemented in the following ways:

[0072] The mapping relationship between inspection images and standard images is reflected in the naming of image files. The naming of inspection images includes the time and location information of the shooting, while the naming of standard images can only include location information. Examples of naming are as follows:

[0073] Standard images: 1#-abc, 2#-abb, etc.

[0074] Inspection images: 20230101-1#-abc, 20230101-2#-abb, etc.;

[0075] The standard image corresponding to inspection image 20230101-1#-abc is 1#-abc, and the standard image corresponding to inspection image 20230101-2#-abb is 2#-abb. Therefore, any inspection image can be used to obtain a unique standard image through its location information.

[0076] For training sample annotation, components and defects are labeled on the image using rectangular bounding boxes to complete the construction of the training samples. Standard images require component annotation using LabelImg, but the annotation specifications differ from those for the inspection images used to construct training. Specifically, standard images do not need to annotate all component targets appearing in the image; only key targets are annotated. Key targets refer to those of interest for subsequent tasks based on component recognition (such as defect detection). These targets have a clear field of view at the acquisition point and are not located in the image edge areas to prevent them from disappearing from the field of view due to positioning errors by the robot and robotic arm. It should be noted that key and non-key targets refer to individual components; therefore, a component type may have both key and non-key targets in the same standard image.

[0077] For fine-grained category classification and detection of component anomalies, this method cascades a classification network after the target recognition network. The classification network uses the bounding boxes of the categories of interest predicted by the component recognition network as regions of interest, performs secondary classification, and outputs more refined categories. For example, the fine-grained categories for bolts can be normal bolts, loose bolts, missing bolts (bolt holes), and non-bolts.

[0078] In summary, the method in this embodiment addresses two types of defects: missing components and fine-grained classification of component anomalies. It designs a robot inspection component defect detection method with two defect diagnosis branches for diagnosis, which can effectively utilize the component identification results to perform targeted defect diagnosis on each component, greatly improving the efficiency of defect diagnosis.

[0079] Specifically, for component loss defects, this method uses an iterative nearest-point method combining category constraints and scale transformation to register the 2D semantic point clouds of the standard and test images, obtaining isolated points in the standard image point cloud to diagnose component loss. Compared to other component loss defect detection methods, this method effectively solves the problems of mismatch in component registration and errors in the position of captured images by incorporating category constraints and scale transformation. For detailed classification of component anomalies, under the condition of lacking training samples, this method designs a method based on random pruning to generate training samples for the classification network, broadening the distribution of classification samples and effectively improving the training effect and prediction accuracy of the classification network.

[0080] As a preferred embodiment, the above component identification results are obtained in the following manner:

[0081] Inspection images of various areas on complex machinery within the workshop are collected. Based on the acquisition location of each inspection image, a standard image is determined using position constraint mapping. A multi-channel prior semantic image of the corresponding area is then obtained from this standard image. The construction of the multi-channel prior semantic image involves: transforming the component annotation information of the standard image to obtain an information form that can express the prior probability of the components on the image; initializing a multi-channel grayscale image with the same number of channels as the total number of all component types in the complex machinery; and superimposing the above information form onto the multi-channel grayscale image to generate the multi-channel prior semantic image, which serves as a quantized prior structural knowledge. The prior semantic image of each channel uses pixel values ​​to represent probabilities. The formal representation defines the prior position and size of a certain component type in the standard image. A pre-constructed multi-channel semantic image input branch is used to fuse the last two layers of highly semantic feature images generated by the trained inspection image feature extraction network for each inspection image with the corresponding multi-channel prior semantic image, thereby enhancing the inspection image features with prior structural knowledge. The aforementioned multi-channel semantic image input branch is constructed by combining the Bottleneck module of ResNet with deformable convolution operators. Based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network, regression and classification are used to obtain the component target recognition result.

[0082] Multi-channel prior semantic images represent the prior semantics (including type, location, and width / height information) of target components in a standard image. To ensure that the multi-channel prior semantic images can correctly modulate features in highly semantic feature images, it is necessary to combine the multi-channel prior semantic images with the low-resolution, highly semantic feature images output by the feature extraction network, thereby improving the accuracy of target recognition. Therefore, this is a component target recognition method that integrates prior structural knowledge.

[0083] This method first acquires inspection images at different locations. Based on positional constraints, a pre-acquired standard image is determined for each inspection image to establish prior semantic structure knowledge corresponding to the acquisition location and viewpoint of each inspection image. Subsequently, this method employs an inspection image feature extraction network to extract features from each inspection image. Simultaneously, a multi-channel semantic image input branch is introduced to adjust the last two layers of highly semantic feature images obtained from the inspection image feature extraction network. Prior semantic structure knowledge is integrated into these two layers of highly semantic feature images before being input into subsequent regression and classification networks. This enhances the inspection image features through prior structural knowledge, effectively improving the accuracy of component recognition.

[0084] The preferred method for information transformation is Gaussian probability elliptic semantic information extraction. Gaussian probability elliptic semantics indicates information such as the length, width, and center point of the target, ensuring that a complete prior semantic distribution is extracted and the distribution curve is smooth.

[0085] As a preferred implementation, the similarity transformation can be calculated using an improved iterative nearest-point method, such as... Figure 2 As shown, the specific calculation method is as follows:

[0086] Execute the CS-ICP method: such as Figure 3 As shown, an initial similarity transformation is used to transform the two-dimensional standard semantic point cloud. For each point in the transformed two-dimensional standard semantic point cloud, nearest neighbor matching with category constraints is performed to find the corresponding point in the two-dimensional test result semantic point cloud, resulting in multiple sets of matching points. Through the multiple sets of matching points, a new similarity transformation is solved. Based on the new similarity transformation, the above process is iterated until the iteration termination condition is reached, resulting in the first similarity transformation.

[0087] The CS-ICP-mono method is executed as follows: the initial similarity transformation is used to transform the two-dimensional standard semantic point cloud. For each point in the transformed two-dimensional standard semantic point cloud, nearest neighbor matching with category constraints is performed to find the corresponding point in the two-dimensional test result semantic point cloud, resulting in multiple sets of matching points. The matching relationship with the largest distance in the many-to-one matching relationship is removed from the multiple sets of matching points. Then, based on the multiple sets of matching points after the removal operation, a new similarity transformation is solved. Based on the new similarity transformation, the above process is iterated until the iteration termination condition is reached to obtain the second similarity transformation.

[0088] Select a similarity transformation from the first and second similarity transformations: Substitute the first and second similarity transformations into the minimum registration residual f to obtain the corresponding minimum registration residual values; take the similarity transformation corresponding to the smaller minimum registration residual value as the final similarity transformation; wherein, the minimum registration residual f is expressed as:

[0089] [x ti y ti 1] T =A·[x i y i 1] T ;

[0090]

[0091]

[0092] In the formula, A represents the similarity transformation, and the first and second similarity transformations are substituted into A respectively; x i ,y i This represents the two-dimensional coordinates of a point in a two-dimensional standard semantic point cloud, (x... di ,y di ) represents point (x) i ,y i ) nearest neighbor matching point.

[0093] In other words, this embodiment adds a scale degree of freedom to the similarity transformation matrix calculated by the Iterative Closest Point (ICP) method, which has three degrees of freedom for planar translation and rotation. At the same time, it constrains each point by its category to obtain the CS-ICP method. Based on CS-ICP, the CS-ICP-mono method is proposed, which removes the larger-distance matching relationships in the many-to-one matching relationships after step two of CS-ICP. In actual diagnosis, the CS-ICP method obtains a more accurate similarity transformation when there is no missing parts, while the CS-ICP-mono method obtains a more accurate similarity transformation when there are missing parts. Therefore, it is necessary to select the optimal similarity transformation from the similarity transformations obtained by these two improved methods through corresponding indicators.

[0094] Regarding the determination of similarity transformations, further relevant explanations are as follows:

[0095] Similarity transformations are typically solved using two methods: singular value decomposition and nonlinear optimization. This embodiment employs the nonlinear optimization method. The homogeneous matrix of the two-dimensional similarity transformation is shown in equation (1):

[0096]

[0097] Where s represents the scale factor, θ represents the rotation angle, and t x and t y This represents the translation of the x-axis and y-axis. The parameters P to be solved for nonlinear optimization are thus constructed, as shown in formula (2):

[0098] P = [t] x t y θs] T (2)

[0099] For a certain point (x) in a two-dimensional standard semantic point cloud (i.e., src point cloud) i ,y i ) and the semantic point cloud of the two-dimensional test results (i.e., the nearest neighbor matching point (x) in the dst point cloud). di ,y di First, apply the current transformation T to (x) i ,y i ) to obtain the transformed point (x) ti ,y ti As shown in formula (3):

[0100] [x ti y ti 1] T =A·[x i y i 1] T (3)

[0101] The Euclidean distance is used as the residual to construct the objective function f, where f is shown in Equation (5):

[0102]

[0103]

[0104] Since the semantic point clouds of the inspection images and standard images are very sparse, the trust region method is used to obtain the accurate global optimum. The trust region method uses the idea of ​​constructing local model sub-problems to gradually approximate the global optimum. The sub-problems obtain the parameters P to be solved by constructing a minimization approximation function, which is shown in Equation (6):

[0105]

[0106] Where G is the first-order partial derivative of the objective function f with respect to each term in the parameter set P, i.e., the Jacobi matrix. H is the second-order partial derivative of G with respect to each term in P, i.e., the Hessian matrix.

[0107] Because the annotations of standard images are incomplete, test semantic point clouds typically have more redundancy than standard semantic point clouds. Therefore, when using the CS-ICP registration method, the standard semantic point cloud is used as the src point cloud, and the test semantic point cloud is used as the dst point cloud. During nearest neighbor matching, the nearest dst point is searched for the src point. That is, it is permissible for dst points not to form a match, but if a src point is needed, it should be matched with a dst point as much as possible. Src points that fail to form a match indicate the loss of a component.

[0108] Furthermore, src and dst may form a many-to-one matching relationship. When a part is missing in the test image, there may be isolated points in the src point cloud. If there are other similar points in dst, when matching according to semantic category, the isolated points will form an incorrect matching relationship with similar dst points. The incorrect matching relationship may affect the accuracy of calculating the similarity transformation. Therefore, based on CS-ICP, this embodiment proposes the CS-ICP-mono method, that is, after the nearest neighbor matching step of CS-ICP, the matching relationship with the larger distance in the many-to-one matching relationship is eliminated. The CS-ICP method and the CS-ICP-mono method are suitable for different scenarios. When there is no missing part, or when the relative change in shooting position between the test image and the standard image is greater, the CS-ICP method is still more accurate. In actual diagnosis, the similarity transformation obtained by these two improved methods can be obtained by substituting the similarity transformation A into formula (5) to obtain the corresponding minimized registration residual; the optimal similarity transformation is selected by comparing the minimized registration residual.

[0109] After registering the test semantic point cloud and the standard semantic point cloud using the optimal similarity transformation A, isolated standard semantic points are extracted as key points to generate the component loss defect target. The specific method can be as follows:

[0110] (1) Apply similarity transformation A to the standard semantic point cloud std_pc to obtain std_pc_A;

[0111] (2) Set the maximum distance threshold threshold = 200, and find the matching point in the test semantic point cloud test_pc with the smallest distance and less than threshold for each std_pc_A point;

[0112] (3) Compare the matching points of each std_pc_A point pairwise. Set the std_pc_A points with the same matching points and larger distances as invalid matching points. For the invalid matching points, find the point with the smallest distance and the same type within the threshold range as the matching point.

[0113] (4) All std_pc_A points that fail to match are considered as key points of component loss. A bounding box with the std_pc_A point as the center is generated with the length and width being 1.5 times the length and width of the standard icon annotation box corresponding to the point, which is used as the final component loss target.

[0114] As a preferred implementation, when the above-mentioned component abnormality fine category classification network outputs fine category results for normal bolts or loose bolts, a 3D point cloud measurement algorithm is used to further confirm whether the bolts are indeed loose. That is, after the bolt category is classified by the classification network, the 3D point cloud plane is segmented twice using the Ransac method to calculate the bolt height in order to improve the accuracy of bolt loosening defect diagnosis. Specifically:

[0115] The 3D camera generates corresponding 3D point cloud information for each image. The predicted bounding boxes of bolts of the normal or loose class obtained by the classification network are mapped to the 3D point cloud to obtain the point cloud information corresponding to the bolt area. The first plane is segmented by the Ransac algorithm with normal vector constraints on the bolt point cloud. Then, the points in the first plane are excluded, and the second plane is segmented by the Ransac algorithm. The first plane and the second plane are set as the top and bottom planes of the bolt.

[0116] After obtaining the plane parameters, the plane distance is calculated using an approximation method: calculate the distance from all points in the second plane to the first plane, sort the distances, and take the average of the set distance intervals as the distance from the second plane to the first plane, which is the height of the bolt.

[0117] The high-precision 3D camera mounted on the end effector of the inspection robot's robotic arm can simultaneously output RGB images and 3D point cloud information. It projects an image encoded according to certain rules and patterns onto the object being measured via an optical engine. The encoded pattern deforms due to modulation of the object's surface shape. The deformed structured light is captured by a fixed-position area array camera. The 3D reconstruction of the object is performed by analyzing the positional relationship between the area array camera and the light source, as well as the degree of structured light deformation. Simultaneously with generating the 3D point cloud, a depth information image matching the RGB image is also generated.

[0118] Specifically, the Ransac method with normal vector constraints follows this process:

[0119] (1) Input the point cloud information corresponding to the bolt region of the first plane point and the first plane normal vector (i.e., constraint normal vector NVcons) after removing the first plane point;

[0120] (2) Randomly select 3 points from the point cloud to determine a plane, and calculate the normal vector NVcur of the random plane;

[0121] (3) Use the inner product to calculate the cos value of the angle between NVcons and NVcur. If the cos value of the angle is greater than the preset threshold of 0.9, the constraint condition is met, indicating that the planes of the two normal vectors are approximately parallel.

[0122] (4) If the constraint is not satisfied, discard the plane directly and return to (2); otherwise, classify all points whose distance from the random plane is less than the threshold as points in the plane.

[0123] (5) Iterate several times and take the random plane with the most points in the plane as the largest plane obtained by partitioning.

[0124] After obtaining the planar parameters, the distance between the two planes needs to be calculated as the bolt height. However, the two divided planes cannot be perfectly parallel, so an approximate method is used to calculate the planar distance. The distance from all points in the second plane to the first plane is calculated, and after sorting the distances, the average of the middle 30% to 70% distances is taken as the distance from the second plane to the first plane, i.e., the bolt height.

[0125] As a preferred embodiment, the training samples used to train the above classification network include samples constructed in the following manner:

[0126] Each bounding box in each training inspection image is used as a cropping box to directly obtain the positive sample with a fine class confidence of 1.0 corresponding to that bounding box;

[0127] The maximum and minimum areas, as well as the maximum and minimum aspect ratios, of each component on the inspection image are statistically analyzed. Several cropping boxes are randomly set for each fine category of each component on the inspection image. First, the center point of the cropping box is obtained by sampling in a uniform distribution on the image area. Then, the area of ​​the cropping box is obtained by sampling in a uniform distribution between the statistically analyzed maximum and minimum areas. Finally, the aspect ratio of the cropping box is obtained by sampling in a uniform distribution between the statistically analyzed maximum and minimum aspect ratios. The specific length and width values ​​are calculated by using the aspect ratio and area to obtain the random cropping box.

[0128] If each cropping box and a corresponding fine-class bounding box in the inspection image have overlapping areas, the Intersection over Union (IoU) ratio between the bounding box and the cropping box is calculated. If each cropping box and multiple bounding boxes of the corresponding fine-class bounding box in the inspection image have overlapping areas, the maximum value of the IoU is taken after calculation. If each cropping box and all bounding boxes of the corresponding fine-class bounding box in the inspection image have no overlapping areas, the IoU is set to 0. If the IoU is greater than or equal to a preset value, the cropping box region is used as a positive sample with a confidence level of 1.0 for the corresponding fine-class; otherwise, the cropping box region is used as a negative sample with a confidence level of (1-IoU).

[0129] The number of positive and negative samples is counted, and additional sampling is used to balance the samples, thus completing the generation of sample labels.

[0130] This method of constructing training samples can expand the distribution range of samples and enhance the robustness of the classification network.

[0131] For bolt type samples, the training samples for the classification network also include samples obtained through the following enhancement methods:

[0132] Step 1: For each fine category of the classification training samples, the training set and test set are split according to a specific ratio;

[0133] Step 2: Augment the dataset samples by performing fine-grained category augmentation on the components in the training and test sets according to the set augmentation factor.

[0134] In other words, after obtaining the training samples for classification, the samples need to be augmented to expand the sample size. Since the ratio of normal, loose, and missing bolt bounding boxes in the dataset is approximately 50:4:1, offline augmentation methods with different augmentation factors are used to expand the training samples to balance the sample size. These offline augmentation methods include: translation transformation, rotation transformation, scaling transformation, Gaussian blur, hue adjustment, brightness adjustment, saturation adjustment, RGB channel pixel value adjustment, and noise addition. These data augmentation methods are adapted to the actual situation during inspection image acquisition, including: addressing positioning errors of the robot and robotic arm through translation, rotation, and scaling transformation; addressing slight shaking of the robotic arm during photography through Gaussian blur; addressing dynamic changes in ambient light and camera exposure through hue, brightness, saturation, and RGB pixel adjustment; and addressing camera background noise through the addition of Gaussian and salt-and-pepper noise. The specific range of each augmentation parameter is determined based on observations of the actual collected samples, as shown in Table 1. Each fine category of the classification training samples is divided into training and test sets at a 4:1 ratio. Simultaneously, the enhancement factors for bolt-normal, loose, and missing bolt-type samples on the training and test sets are 2, 5, and 20, respectively; ensuring that the number of samples in each abnormal category after augmentation is not less than 20% of the normal category. Negative class samples are not augmented.

[0135] Table 1 Data Augmentation Parameter Range Settings

[0136]

[0137] After training the classification network using the dataset created in the above manner, a well-trained classification network can be obtained. The bolt height is calculated by segmenting the 3D point cloud plane using the Ransac method twice to separate the bolt and loose bolt categories from the classification results, thereby improving the detection accuracy of bolt loosening defects.

[0138] As a preferred implementation, this embodiment proposes different measurement methods for the diagnosis of over-limit defects, taking into account the different characteristics of the two components: brake pads and stone sweeper height.

[0139] For brake pad thickness measurement, the acquired images usually show multiple brake pads. Since they are tightly pressed against the wheel axle and brake disc by the brake calipers during braking, their wear is uniform; therefore, only the thickness of one needs to be measured. To improve measurement accuracy, the outermost brake pad with the clearest field of view is usually selected. Brake pad images have obvious upper and lower edges; therefore, this embodiment preferably uses an edge detection algorithm to detect the edges. The specific implementation is as follows:

[0140] Step 1: Extract the image region of the gate and initialize the two thresholds as (250, 180);

[0141] Step 2: Perform Canny edge detection using the current dual thresholds to obtain the edge response;

[0142] Step 3: Record the coordinates of the edge points on the vertical centerline (which is parallel to the Y-axis). At the midpoint of the X-axis (horizontal direction of the image) of the edge response, record the coordinates of the edge points from top to bottom along the Y-axis.

[0143] Step 4: If there are at least two edge points on the vertical centerline, proceed to Step 5 to calculate the thickness; otherwise, reduce both thresholds by 60 and return to Step 2.

[0144] Step 5: Using the vertical centerline edge points with the smallest and largest Y values ​​as the upper and lower edge points of the brake pad, respectively, map the coordinates of the upper and lower edge points to the 3D point cloud generated by the camera to obtain the Y values ​​in the real-world coordinate system. Use the difference between the Y values ​​of the upper and lower edge points as the final thickness measurement result.

[0145] The Canny algorithm used in this method only needs to meet the requirements of brake pad measurement; it is not necessary to obtain a clear edge. Therefore, as long as at least two edge points on the vertical central axis are obtained, the brake pad thickness measurement can be achieved.

[0146] For measuring the height of a stone sweeper, since the background in the field of view of the stone sweeper is relatively simple and the straight line characteristics of the track plane are obvious, the inspection image can be preprocessed and then a measurement baseline can be obtained based on straight line fitting to obtain the height value.

[0147] The image preprocessing procedure for the stone sweeper is as follows: the entire image is converted to grayscale, and a segmentation threshold of 18 is set. Figure 2 Binarization. After obtaining the corresponding binarized image, a 5×5 median filter is used to initially remove isolated black and white noise. A morphological opening operation with a kernel size of 3 is used to further remove white noise. A morphological closing operation with a kernel size of 11 is used to further remove black noise, resulting in a smoothed binary image. The preprocessing process then ends.

[0148] Furthermore, on the smoothed binary image, based on the sweeper bounding box predicted by the component recognition network, the boundary line between the sweeper and the track plane is found. The process is as follows:

[0149] Step 1: Map the bounding box of the stone sweeper onto the binary image. Find the first black pixel point downwards from the center point of the stone sweeper bounding box (i.e., the positive Y-axis direction of the image) as the lower boundary of the stone sweeper. Then continue downwards to find the first white pixel point as the upper boundary of the track plane. Add these two boundary points (coordinate information) and their height difference H to the set Thickness (i.e., the height difference set).

[0150] Step 2: Move left and right along the X-axis from the center point of the sweeper, moving 2 coordinates each time, and then repeat Step 1.

[0151] Step 3: Let Step 2 traverse the range of the X-axis ±50 coordinates of the center point of the sweeper. Finally, there are 51 sets of boundary points and height difference H in Thickness.

[0152] Step 4: Sort the Thicknesses according to their height difference H, and remove the two sets of boundary points with the smallest and largest H, leaving 49 sets.

[0153] Step 5: Remove outliers using the mean and median of the height differences H. Calculate the mean (Mean) and median (Median) of H. If the height difference H of a set of boundary points meets the following criteria... or The boundary points in this group are then considered outliers and removed from Thickness.

[0154] Step Six: Take the smaller quartile index (Quarter_index) after sorting by height difference H in the current Thickness, and use the boundary points corresponding to the indices within the range of Quarter_index ± 5 to fit the straight line of the lower boundary of the sweeper and the straight line of the upper boundary of the track plane using the least squares method. Average the slopes of the two lines to obtain a uniform fitting slope k.

[0155] Step 7: The upper boundary point corresponding to the quartile index (Quarter_index) is (x0, y ... top The lower boundary point is (x0, y0). bottom ), passing through these two points respectively, and fitting straight lines with the same upper and lower boundaries using the slope k, with the intercepts as shown in formula (7):

[0156]

[0157] Step 8: Calculate the distance between the fitted straight lines of the final upper and lower boundaries, and use it as the final height of the sweeper from the track surface.

[0158] For example, the midpoint of the upper and lower boundary points corresponding to Quarter_index. The foot of the perpendicular f to the fitted line at the final upper and lower boundaries top and f bottom As the two points used to calculate the distance, their coordinates are shown in formula (8):

[0159]

[0160] f top and f bottom Mapping to the 3D point cloud yields the Y-axis height in the real-world coordinate system, and the difference is used to obtain the height of the sweeper from the track surface.

[0161] The classification network proposed in this application uses the ResNet50 network. Other network models, such as AlexNet, VGG16, and Inception-ResNet-v2, can also replace the method in this application.

[0162] Example 2

[0163] A robot inspection component defect detection device includes: a computer-readable storage medium and a processor;

[0164] The computer-readable storage medium is used to store executable instructions;

[0165] The processor is used to read executable instructions stored in the computer-readable storage medium and execute a robot inspection component defect detection method as described in Embodiment 1.

[0166] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0167] Example 3

[0168] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to perform a robot inspection component defect detection method as described in Embodiment 1.

[0169] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0170] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of robotically inspecting a component for defects, the method comprising: The method comprises the following steps: Converting the component recognition result corresponding to each inspection image obtained by the target recognition network into a two-dimensional test result semantic point cloud; Based on the collection position of each inspection image, a standard image is determined by using position constraint mapping, all the annotation boxes in the standard image are converted into a two-dimensional standard semantic point cloud, the similarity transformation of the two point clouds is determined to perform point cloud registration, the distance between each point in the two-dimensional standard semantic point cloud and the corresponding point in the two-dimensional test result semantic point cloud is calculated, and if the distance is greater than a threshold, the point is regarded as an isolated point; the annotation box information corresponding to the isolated point is regarded as the type and position of the missing component, and the component missing defect diagnosis is completed; And, Each bounding box in the component recognition result of each inspection image is cropped and input into a component abnormal fine category classification network to obtain the component abnormal fine category result corresponding to each bounding box in the inspection image; wherein the training samples for training the classification network include samples obtained by the following method: based on the annotation box of each component in the training inspection image, a plurality of regions are randomly cropped on the image to obtain fine category training samples of the component, and the fine category includes a normal category, an abnormal category and a non-component background category; The component recognition result is obtained by the following method: Inspection images of each region on complex machines in a workshop are collected, based on the collection position of each inspection image, a standard image is determined by using position constraint mapping, and a multi-channel prior semantic image of the corresponding region obtained from the standard image is obtained, wherein the multi-channel prior semantic image is constructed in the following manner: the component annotation information of the standard image is converted into an information form capable of expressing the prior probability of the component on the image; a multi-channel gray image with the number of channels consistent with the total number of all component types in the complex machine is initialized; the information form is superimposed on the multi-channel gray image to generate a multi-channel prior semantic image as a tensorized prior structural knowledge, and each channel of the prior semantic image represents the prior position and size of a certain component type in the standard image in the form of probability represented by pixel value; The last two layers of high semantic feature images generated by the trained inspection image feature extraction network and the multi-channel prior semantic image corresponding to each inspection image are fused by using a pre-constructed multi-channel semantic image input branch to enhance the prior structural knowledge on the inspection image features; wherein the multi-channel semantic image input branch is constructed by combining the Bottleneck module of Resnet and the deformable convolution operator; Based on the fused feature images and the features output by other output layers of the inspection image feature extraction network, the component target recognition result is obtained by regression and classification.

2. The method of claim 1, wherein, The similarity transformation is calculated by using an improved iterative closest point method, and the specific calculation method is as follows: Transforming the two-dimensional standard semantic point cloud by using an initial similarity transformation, performing nearest neighbor matching with class constraints for each point in the transformed two-dimensional standard semantic point cloud, finding the corresponding points in the two-dimensional test result semantic point cloud, and obtaining a plurality of groups of matching points; solving a new similarity transformation through the plurality of groups of matching points; iterating the above process based on the new similarity transformation until an iteration termination condition is reached, and obtaining a first similarity transformation; Transforming the two-dimensional standard semantic point cloud by using the initial similarity transformation, performing nearest neighbor matching with class constraints for each point in the transformed two-dimensional standard semantic point cloud, finding the corresponding points in the two-dimensional test result semantic point cloud, and obtaining a plurality of groups of matching points; eliminating the matching relationship with the largest distance in the many-to-one matching relationship from the plurality of groups of matching points, and then solving a new similarity transformation according to the plurality of groups of matching points after the elimination operation; iterating the above process based on the new similarity transformation until an iteration termination condition is reached, and obtaining a second similarity transformation; applying the first similarity transformation and the second similarity transformation into the minimization registration residual respectively , obtaining corresponding minimization registration residual values; taking the similarity transformation corresponding to the smaller minimization registration residual value as the final similarity transformation; wherein the minimization registration residual is expressed as: ； ； ； wherein A represents a similarity transformation, the first similarity transformation and the second similarity transformation being respectively A substituted; represents a two-dimensional coordinate of a certain point in the two-dimensional standard semantic point cloud, represents the nearest neighbor matching point of the point ( ).

3. The method of claim 1, wherein, When the component abnormal fine category classification network outputs a fine category result of a normal bolt and / or a loose bolt, a 3D point cloud measurement algorithm is used to further confirm whether the bolt is indeed loose, specifically: Mapping the bounding box corresponding to the fine category result of the normal bolt and the loose bolt in the component recognition result to the 3D point cloud to obtain point cloud information corresponding to the bolt region; wherein the 3D point cloud is 3D point cloud information generated correspondingly when each inspection image is collected; Using a Ransac algorithm to segment a first plane from the point cloud information corresponding to the bolt region, and calculating a normal vector of the first plane; excluding the points of the first plane from the point cloud information corresponding to the bolt region, and continuing to segment a second plane using a Ransac method with a first plane normal vector constraint, the first plane and the second plane being set as the top and bottom planes of the bolt; After obtaining the first plane and the second plane, the plane distance is calculated using an approximation method, specifically, the distance from all points in the second plane to the first plane is calculated, and after sorting the distance, the mean value of the distance interval is taken as the distance from the second plane to the first plane, that is, the height of the bolt.

4. The method of claim 3, wherein, The Ransac method with a first plane normal vector constraint is specifically: S1, randomly selecting three points from the point cloud information corresponding to the bolt region after excluding the points of the first plane to determine a plane, and calculating the normal vector of the random plane; S2, calculating the included angle cos value between the first plane normal vector and the normal vector of the random plane using inner product, if the included angle cos value is greater than a preset threshold, the constraint condition is satisfied, indicating that the two normal vectors are approximately parallel, and all points with a distance less than the threshold from the random plane are divided into in-plane points; if the constraint is not satisfied, the random plane is discarded directly, and step S1 is returned until the iteration is performed for a preset number of times; S3, taking the random plane with the most in-plane points as the maximum plane segmented, that is, the second plane.

5. The method of claim 1, wherein, The training samples for training the classification network include samples constructed in the following manner: Taking each annotation box of each training inspection image as a cropping box to directly obtain a positive sample with a fine category confidence of 1.0 corresponding to the annotation box; The maximum and minimum areas of each component on the image are counted, and the maximum and minimum aspect ratios are counted. A plurality of cutting frames are randomly set for each fine category of each component on the image, that is, the center points of the cutting frames are sampled on the image region in a uniform distribution, the areas of the cutting frames are sampled in a uniform distribution between the counted maximum and minimum areas, and the aspect ratios of the cutting frames are sampled in a uniform distribution between the counted maximum and minimum aspect ratios. The specific length and width are calculated through the aspect ratio and the area, and the random cutting frame is obtained. If each cutting frame and a certain labeled frame corresponding to the fine category on the image have an overlapping region, the intersection-over-union (IoU) of the labeled frame and the cutting frame is calculated. If each cutting frame and a plurality of labeled frames corresponding to the fine category on the image have overlapping regions, the maximum value of the IoU is taken after the calculation. If each cutting frame and each labeled frame corresponding to the fine category on the image have no overlapping region, the IoU is set to 0. If the IoU is greater than or equal to a preset value, the cutting frame region is taken as a positive sample with a confidence of 1.0 for the corresponding fine category, otherwise, the cutting frame region is taken as a negative sample with a confidence of (1-IoU) for the negative category. The number of positive and negative samples is counted, and the sample is balanced through additional sampling to complete the generation of sample labeling.

6. The method of claim 1, wherein, The inspection image is an inspection image of a rail transit vehicle. The method further comprises detecting the thickness of the brake pad on the rail transit vehicle to determine whether the brake pad is out of limit, specifically: Step one, intercepting the image region of the brake pad in the current inspection image, and initializing the double threshold value as (250, 180); Step two, performing Canny edge detection using the current double threshold value to obtain an edge response; Step three, recording the edge point coordinates on the vertical center axis in the image region, that is, recording the coordinates of the edge points from top to bottom in the Y-axis direction at the X-axis midpoint of the edge response; Step four, if the number of edge points on the vertical center axis is not less than two, proceed to step five to calculate the thickness; otherwise, reduce the double threshold value by a preset value and return to step two; Step five, taking the vertical center axis edge points with the minimum and maximum Y values as the upper and lower edge points of the brake pad, respectively, mapping the coordinates of the upper and lower edge points to the 3D point cloud generated when the camera collects the inspection image to obtain the Y value in the real world coordinate system, and taking the difference between the Y values of the upper and lower edge points as the final thickness measurement result.

7. The method of claim 1, wherein, The inspection image is an inspection image of a rail transit vehicle. The method further comprises detecting the height of the stone sweeper on the rail transit vehicle to determine whether the stone sweeper is out of limit, specifically: The inspection image containing the stone sweeper is preprocessed to obtain a balanced binary image; On the smoothed binary image, based on the boundary box of the stone sweeper predicted by the target recognition network, the boundary lines of the stone sweeper and the track plane are found, and the process is as follows: S1, mapping the stone sweeper bounding box to the binary image; on the binary image, finding the first black pixel point vertically downward from the center point of the stone sweeper bounding box as the lower boundary of the stone sweeper, then continuing to find the first white pixel point as the upper boundary of the track plane, and adding the coordinate information of the two boundary points and the height difference H to the height difference set; S2, offsetting left and right from the center point of the stone sweeper bounding box to the X-axis direction, moving 2 coordinates each time, and then repeating step one, wherein the coordinate range of left and right offset is set, and finally the height difference set has multiple sets of boundary point coordinate information and height difference H; S3, sorting the height difference set according to the height difference H, and removing the two sets with the smallest and largest H to obtain a new height difference set; S4, removing outliers by the mean and median of the height difference H in the new height difference set to obtain a final height difference set; S5, taking the smaller X quantile index X_index in the final height difference set sorted by the height difference H, and fitting the straight line of the lower boundary of the stone sweeper and the straight line of the upper boundary of the track plane with the boundary points corresponding to the index in the range of X_index±A respectively, taking the mean of the slopes of the two straight lines to obtain the unified fitting slope k, wherein X and A are positive numbers; S6、the upper boundary point corresponding to the X_index is , the lower boundary point is , and the intercept of the upper and lower boundary fitting straight lines with the same final slope k is ； S7, based on the intercept, obtaining the distance of the final upper and lower boundary fitting straight line as the final height of the stone sweeper to the track plane.

8. The method of claim 7, wherein, The way to remove outliers in step S4 is: calculate the mean value Mean and the median Median of H, if the height difference H of a group of boundary points meets , this group of boundary points is regarded as outliers, and is removed from the new height difference set. And / or, In step S5, X is four and A is 5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute a robot inspection component defect detection method according to any one of claims 1 to 8.

Images