Metal component surface crack detection method based on visual-haptic fusion perception

Abstract

The application discloses a kind of visual haptics fusion perception's metal component surface crack detection method, comprising: the image of the metal component surface to be detected is collected by RGB-D camera, and the collected image is adaptively image enhanced and denoising pre-processing, obtain the image after pre-processing;From the visual geometric feature vector of crack in the image after pre-processing;Position to crack area by haptic sensor, collect the 3D topography of crack, displacement field and distribution force data, and calculate the haptic measurement depth of crack based on mechanical model, then construct haptic feature vector based on haptic measurement depth;Wavelet transform is used to multiscale alignment of visual geometric feature vector and haptic feature vector, and feature level fusion is carried out by zero neural network optimization algorithm, to obtain fusion feature;Based on fusion feature, use machine learning classifier to carry out crack detection, output crack size, position and confidence level.The application can improve the accuracy and reliability of detection.

Description

Technical Field

[0001] This invention relates to the field of crack detection technology, and more specifically to a method for detecting surface cracks in metal parts using a fusion of visual and tactile perception. Background Technology

[0002] Crack detection in metal components of high-speed railways is a core technical aspect for ensuring the safe operation of rail transit systems, preventing major accidents, and extending equipment lifespan. Cracks, as a major manifestation of metal fatigue damage, can be effectively prevented from fracturing and failing in critical components such as bogies, wheelsets, and brake discs during high-speed operation if identified accurately in their early stages. Traditional detection methods mainly rely on manual visual inspection and non-destructive testing (NDT) techniques, which have significant limitations: 1) Manual inspection requires downtime for maintenance, resulting in a significant conflict between inspection efficiency and the high-frequency operation requirements of trains, and is also susceptible to missed detections due to personnel experience and fatigue; 2) Conventional NDT methods, such as ultrasonic testing, require contact coupling, making it difficult to achieve online monitoring of moving parts; 3) Industrial endoscope inspection is limited by its narrow field of view, limiting its ability to identify cracks within complex structures.

[0003] With the development of intelligent detection technology, crack detection methods based on computer vision are gradually being applied to the rail transit field. However, existing technologies mainly rely on a single sensor for detection and cannot achieve multimodal fusion, so the accuracy and reliability of detection need to be improved. Summary of the Invention

[0004] In view of this, the present invention provides a method for detecting surface cracks in metal parts by integrating visual and tactile perception, so as to give full play to the complementary advantages of cross-modal data and improve the accuracy and reliability of detection.

[0005] A method for detecting surface cracks in metal parts using a fusion of visual and tactile perception includes: Step S1: Acquire an image of the surface of the metal part to be inspected using an RGB-D camera, and perform adaptive image enhancement and noise reduction preprocessing on the acquired image to obtain the preprocessed image; Step S2: Extract the visual geometric feature vector of the crack from the preprocessed image. The visual geometric feature vector includes the crack length, width, visual depth, and visual depth statistical distribution features. Step S3: Locate the crack area using a tactile sensor, collect 3D morphology, displacement field and distributed force data of the crack, calculate the tactile measurement depth of the crack based on the mechanical model, and then construct a tactile feature vector based on the tactile measurement depth. Step S4: Wavelet transform is used to align the visual geometric feature vector and the tactile feature vector at multiple scales, and feature-level fusion is performed through a zero-return neural network optimization algorithm to obtain fused features; Step S5: Based on the fused features, a machine learning classifier is used to detect cracks and output the crack size, location, and confidence level.

[0006] The method for detecting surface cracks in metal parts based on visual-tactile fusion perception provided by the present invention has the following beneficial effects: (1) This invention achieves a significant improvement in detection accuracy through the deep collaboration of visual and tactile sensors. The RGB-D camera provides global crack morphology information, while the tactile sensor provides precise local mechanical parameters, forming an effective complementarity. Experimental data shows that in the test of high-speed rail wheel samples, the F1 score of this invention reaches 95.2%, which is significantly better than the single visual detection method.

[0007] (2) Traditional visual inspection methods are susceptible to changes in lighting and surface texture interference, while this invention effectively overcomes these limitations through a dual verification mechanism. The visual inspection stage employs depth-guided image enhancement technology combined with a multi-scale filtering strategy; the tactile verification stage provides physical-level real data verification through a mechanical model, forming a redundant verification mechanism. When the visual sensor is interfered with by surface reflections or shadows, tactile data provides reliable backup verification; conversely, when the tactile sensor exhibits positioning deviation, visual coordinate calibration can correct the error in real time. This bidirectional fault-tolerant mechanism ensures stable operation of the system in complex industrial environments, enhancing robustness and reliability.

[0008] (3) This invention achieves excellent real-time performance through deep optimization at the algorithm level. Experimental results show that the single-frame processing time of this invention is controlled within 80ms, which fully meets the real-time requirements of less than 100ms in industrial settings and effectively improves detection efficiency. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the method for detecting surface cracks in metal components using a visual-tactile fusion perception system provided in an embodiment of the present invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the present invention, and should not be construed as limiting the present invention.

[0011] Please see Figure 1 The present invention provides a method for detecting surface cracks in metal parts using a visual-tactile fusion perception system, comprising steps S1 to S5: Step S1: Acquire an image of the surface of the metal part to be inspected using an RGB-D camera, and perform adaptive image enhancement and noise reduction preprocessing on the acquired image to obtain the preprocessed image.

[0012] In step S1, the adaptive image enhancement employs a contrast-limited adaptive histogram equalization algorithm, with a contrast limit threshold. The value range is 2.0-4.0. In this embodiment, the contrast limit threshold is... Set the grid size to 3.2. for .

[0013] In step S1, noise preprocessing is performed using a median filtering algorithm. A 5×5 filtering window is used, and the median of the pixel gray value sequence within the window is assigned to the center pixel to suppress and remove image noise.

[0014] Step S2: Extract the visual geometric feature vector of the crack from the preprocessed image. The visual geometric feature vector includes the crack length, width, visual depth, and visual depth statistical distribution features.

[0015] The extraction of visual geometric feature vectors specifically includes two sub-steps: edge detection and texture analysis. The edge detection sub-step uses the Canny operator to extract contours and combines morphological closing operations to connect fracture edges. The texture analysis sub-step calculates the contrast and entropy features of the gray-level co-occurrence matrix to effectively distinguish between real cracks and interference caused by material surface textures.

[0016] Visual geometric feature vectors for:

[0017] in, The length of the crack. The crack width is... Visual depth of the crack The variance of the visual depth of the crack. The standard deviation of the visual depth of the crack is given, with units in mm. Z-score standardization was used to eliminate the influence of dimensions. This indicates transpose.

[0018] Step S3: Locate the crack area using a tactile sensor, collect 3D morphology, displacement field and distributed force data of the crack, calculate the tactile measurement depth of the crack based on the mechanical model, and then construct a tactile feature vector based on the tactile measurement depth.

[0019] Among them, 3D morphology, displacement field and distributed force data of the crack area are collected by tactile sensors at a sampling rate of 1000Hz.

[0020] The tactile sensor's localization is achieved through the CoreXY motion structure, and the kinematic model expression is as follows:

[0021]

[0022] in, , Let the coordinates be the target position coordinates of the motion platform in a two-dimensional plane. and These are the real-time rotation angles of the first and second motors driving the CoreXY motion structure, respectively. In this embodiment, the radius of the drive pulley that directly meshes with the motor's synchronous belt is... It is 5mm.

[0023] The formula for calculating the tactile depth of a crack is:

[0024] in, The tactile depth of the crack is measured in mm. The total number of samples; Indicates the first The height position at the time of sampling is in mm; This is the initial contact position, in mm; Contact force, measured in N; The system stiffness coefficient is calibrated, with units of N / mm, and is determined through calibration experiments. In this embodiment, it is taken as 0.5 N / mm.

[0025] Constructed tactile feature vector The expression is:

[0026] in, , , These are the average values ​​of the contact forces in the x, y, and z directions, respectively. , , These are the variances of the contact forces in the x, y, and z directions, respectively. , , These represent the peak values ​​of the contact forces in the x, y, and z directions, respectively.

[0027] In practice, PCA can be used to compress the 128-dimensional features to 20 dimensions to construct a tactile feature vector. .

[0028] Step S4: Wavelet transform is used to align the visual geometric feature vector and the tactile feature vector at multiple scales, and feature-level fusion is performed through a zero-return neural network optimization algorithm to obtain fused features.

[0029] Among them, wavelet transform is used to perform three-level wavelet decomposition of visual geometric feature vectors and tactile feature vectors, and feature matching is performed at different scales.

[0030] In this embodiment, when performing feature-level fusion using the Zero-Sum Neural Network (ZNN) optimization algorithm, an attention mechanism is introduced for weighting, and the weight calculation formula is as follows:

[0031] in, For the fusion feature of the first Features Attention weights For the fusion feature of the first Features Feature scoring, For the fusion feature of the first Features Feature scoring, This represents the total dimension of the feature vectors before fusion.

[0032] In this embodiment, the zero-return neural network optimization algorithm dynamically solves the following objective function when performing feature-level fusion:

[0033] in, This indicates taking the minimum value. This is a regularization hyperparameter used to adjust the weight of the model complexity penalty term in the objective function; It concerns the fusion parameters to be optimized. Regularization functions are used to constrain... The complexity is reduced to prevent overfitting.

[0034] Among them, fusion parameters Satisfy the following formula:

[0035] in, yes The first-order differential, Indicates partial derivative, It depends on the parameters scalar objective function, Describing the scalar objective function Relative to parameters The gradient matrix, This indicates that the gradient matrix is ​​exponentiated. The convergence coefficient is . To act on the scalar objective function The nonlinear activation function.

[0036] Optimal solution of the objective function The optimal fusion mapping relationship between visual geometric feature vectors and tactile feature vectors is defined. It should be noted that the ZNN optimization algorithm dynamically solves a well-defined feature fusion optimization objective function through its internal dynamic model, thereby automatically learning the optimal cross-modal feature fusion parameters. These parameters are directly used to generate more discriminative fused features, which is a core technical step in improving the accuracy of subsequent crack detection.

[0037] Step S5: Based on the fused features, a machine learning classifier is used to detect cracks and output the crack size, location, and confidence level.

[0038] The confidence level is calculated using the following formula:

[0039] in, Indicates the confidence level. The visual discrimination confidence is based on the visual geometric feature vectors obtained in step S2. The confidence level for mechanical discrimination based on the tactile feature vector obtained in step S3, The fusion discrimination confidence is based on the fusion features obtained in step S4; , , They are respectively , , non-negative weighting coefficients , , The specific values ​​are adaptively determined by training and optimizing the detection model using a dataset of labeled crack samples, so as to achieve optimal consistency between the overall confidence assessment and the actual results.

[0040] In this embodiment, , , The values ​​are 0.4, 0.3, and 0.3 respectively. In this embodiment, the confidence threshold is set to 0.85. The confidence threshold is a clear and executable judgment standard for the final detection. When the confidence calculated by the algorithm is greater than or equal to 0.85, the system will determine that a crack exists and output the specific parameters of the crack, including the crack size and location. When the overall confidence is less than 0.85, it will be determined that there is no crack.

[0041] To verify the effectiveness of the invention, tests were conducted on an experimental platform simulating the actual operation and maintenance environment of high-speed rail. 1. Experimental Setup Test sample: 15 thick steel plates simulating cracks were used, and 32 cracks were confirmed. The crack length ranged from 5 to 40 mm, and the width ranged from 0.1 to 1.5 mm.

[0042] Hardware and software: The RGB-D camera is an OAK-D-Pro, and the haptic sensor is a Tac3D-Mk2 (force resolution 0.01N). The system runs on an Intel i7-11700K CPU, and the software environment is OpenCV 4.5.1 and Python 3.8.

[0043] Comparison method: The present invention is compared with two comparison methods. Comparison method 1 is a single visual detection method based on YOLOv5s, and comparison method 2 is a single tactile detection method based on fixed path scanning.

[0044] Evaluation metrics: Accuracy, recall, precision, and F1 score are used as the main evaluation metrics.

[0045] 2. Performance Comparison Results Table 1 compares the performance metrics of different methods across 1280 independent tests involving 32 cracks (data are the mean of 5-fold cross-validation). Table 1

[0046] As shown in Table 1, the F1 score of this invention reaches 95.2%, and compared with single vision or single tactile methods, the accuracy, recall, and precision of this invention are significantly improved. This fully demonstrates the advantage of multimodal fusion in significantly improving detection reliability in complex industrial scenarios. Meanwhile, the 78ms single-frame processing time meets the industrial real-time requirements (<100ms).

[0047] In summary, the visual-tactile fusion perception method for detecting surface cracks in metal parts according to the above embodiments has the following beneficial effects: (1) This invention achieves a significant improvement in detection accuracy through the deep collaboration of visual and tactile sensors. The RGB-D camera provides global crack morphology information, while the tactile sensor provides precise local mechanical parameters, forming an effective complementarity. Experimental data shows that in the test of high-speed rail wheel samples, the F1 score of this invention reaches 95.2%, which is significantly better than the single visual detection method.

[0048] (2) Traditional visual inspection methods are susceptible to changes in lighting and surface texture interference, while this invention effectively overcomes these limitations through a dual verification mechanism. The visual inspection stage employs depth-guided image enhancement technology combined with a multi-scale filtering strategy; the tactile verification stage provides physical-level real data verification through a mechanical model, forming a redundant verification mechanism. When the visual sensor is interfered with by surface reflections or shadows, tactile data provides reliable backup verification; conversely, when the tactile sensor exhibits positioning deviation, visual coordinate calibration can correct the error in real time. This bidirectional fault-tolerant mechanism ensures stable operation of the system in complex industrial environments, enhancing robustness and reliability.

[0049] (3) This invention achieves excellent real-time performance through deep optimization at the algorithm level. Experimental results show that the single-frame processing time of this invention is controlled within 80ms, which fully meets the real-time requirements of less than 100ms in industrial settings and effectively improves detection efficiency.

[0050] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for detecting surface cracks in metal components using a fusion of visual and tactile perception, characterized in that, include: Step S1: Acquire an image of the surface of the metal part to be inspected using an RGB-D camera, and perform adaptive image enhancement and noise reduction preprocessing on the acquired image to obtain the preprocessed image; Step S2: Extract the visual geometric feature vector of the crack from the preprocessed image. The visual geometric feature vector includes the crack length, width, visual depth, and visual depth statistical distribution features. Step S3: Locate the crack area using a tactile sensor, collect 3D morphology, displacement field and distributed force data of the crack, calculate the tactile measurement depth of the crack based on the mechanical model, and then construct a tactile feature vector based on the tactile measurement depth. Step S4: Wavelet transform is used to align the visual geometric feature vector and the tactile feature vector at multiple scales, and feature-level fusion is performed through a zero-return neural network optimization algorithm to obtain fused features; Step S5: Based on the fused features, a machine learning classifier is used to detect cracks, and the crack size, location, and confidence level are output. In step S2, the visual geometric feature vector for: in, The length of the crack. The crack width is... Visual depth of the crack The variance of the visual depth of the crack. The standard deviation of the visual depth of the crack. Indicates transpose; In step S3, the formula for calculating the tactile depth of the crack is: in, To measure the depth of a crack by touch. The total number of samples, Indicates the first Height position at the time of the next sampling This is the initial contact position. For contact force, The system stiffness coefficient is calibrated. In step S3, the tactile feature vector The expression is: in, , , These are the average values ​​of the contact forces in the x, y, and z directions, respectively. , , These are the variances of the contact forces in the x, y, and z directions, respectively. , , These represent the peak values ​​of the contact forces in the x, y, and z directions, respectively. In step S4, when performing feature-level fusion using the zero-return neural network optimization algorithm, an attention mechanism is introduced for weighting. The weight calculation formula is as follows: in, For the fusion feature of the first Features Attention weights For the fusion feature of the first Features Feature scoring, For the fusion feature of the first Features Feature scoring, This represents the total dimension of the feature vectors before fusion. In step S4, when the zero-return neural network optimization algorithm performs feature-level fusion, it dynamically solves the following objective function: in, This indicates taking the minimum value. For regularization hyperparameters, It concerns the fusion parameters to be optimized. The regularization function; Fusion parameters Satisfy the following formula: in, yes The first-order differential, Indicates partial derivative, It depends on the parameters scalar objective function, Describing the scalar objective function Relative to parameters The gradient matrix, This indicates that the gradient matrix is ​​exponentiated. The convergence coefficient is . To act on the scalar objective function The nonlinear activation function.

2. The method for detecting surface cracks in metal parts using visual-tactile fusion perception according to claim 1, characterized in that, In step S1, adaptive image enhancement employs a contrast-limited adaptive histogram equalization algorithm, with a contrast limit threshold. The value range is 2.0-4.0, and the grid size is [not specified]. for ; In step S1, noise reduction preprocessing is performed using a median filtering algorithm. A 5×5 filtering window is used, and noise reduction is achieved by assigning the median of the pixel grayscale value sequence within the window to the center pixel.

3. The method for detecting surface cracks in metal parts using visual-tactile fusion perception according to claim 1, characterized in that, In step S5, the confidence level is calculated using the following formula: in, Indicates the confidence level. The visual discrimination confidence is based on the visual geometric feature vectors obtained in step S2. The confidence level for mechanical discrimination based on the tactile feature vector obtained in step S3, The fusion discrimination confidence is based on the fusion features obtained in step S4; , , They are respectively , , The non-negative weighting coefficients.

Images