A surface defect detection model based on tactile feedback and a construction method and detection method thereof
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2025-04-11
- Publication Date
- 2026-06-26
AI Technical Summary
Existing computer vision-based surface defect detection methods cannot work in narrow or enclosed environments and are easily affected by lighting and material surface characteristics, making it difficult to achieve efficient and high-precision detection of minute defects.
A surface defect detection model based on tactile feedback is adopted. The tactile signals of the product surface are trained by a convolutional neural network, and a feature extractor and classifier are constructed by a class-guided contrastive learning strategy to achieve high-precision detection of subtle defects.
It achieves the capture of morphological features at the 10-micron level, is applicable to transparent, highly reflective and complex textured materials, improves detection accuracy and applicability, and overcomes the limitations of vision methods.
Smart Images
Figure CN120449002B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of product surface defect detection technology, and more specifically, relates to a surface defect detection model based on tactile feedback, its construction method, and detection method. Background Technology
[0002] High-quality surfaces are crucial for ensuring product performance and lifespan. Many cutting-edge manufacturing industries have stringent requirements for product surface quality, such as aircraft manufacturing, nuclear fuel container manufacturing, and artificial organ manufacturing. To ensure product durability and safety, the size of surface defects in these industries is typically limited to 0.01 mm. Therefore, to improve finished product quality and eliminate safety hazards, the manufacturing process must include surface quality inspection.
[0003] Manual inspection was a common surface quality inspection method in early industrial fields, including visual inspection, contact inspection, and optical inspection. These methods relied on manpower and were inefficient. As modern industry's demands for efficiency, precision, and automation have increased, the shortcomings of traditional methods have become increasingly apparent. In recent years, the development of computer vision has provided an automated, efficient, and low-cost solution for product surface defect inspection, attracting widespread attention from academia and industry. It is currently applied in many manufacturing fields, such as steel stamping, ceramic sintering, automotive body welding, and printed circuit boards.
[0004] However, when dealing with the task of detecting minute defects, this type of visual image-based method has the following limitations: ① It is constrained by the physical characteristics of optical sensors, and cannot work in narrow or enclosed low-light conditions, nor in irradiated environments; ② The imaging results are easily affected by the surface curvature, texture, and reflection of the material product, and the imaging results of minute defects may differ at different illumination angles. Therefore, there is an urgent need to develop a surface minute defect detection method that is highly adaptable to different working conditions, efficient, and accurate. Summary of the Invention
[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a surface defect detection model based on tactile feedback, its construction method and detection method, the purpose of which is to achieve high-precision detection of minute defects on the product surface.
[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for constructing a surface defect detection model based on tactile feedback is proposed, comprising the following steps:
[0007] The convolutional neural network model is trained using tactile signals from the product surface and their defect category labels. This model includes a feature extractor and a classifier. During training:
[0008] The feature extractor takes the tactile signal of the product surface and its defect category label as input. It adopts a class-guided contrastive learning strategy to extract discriminative features that can distinguish the signal category from the tactile signal and compress them into a representation vector as output. The classifier takes the representation vector and the corresponding defect category label as input and outputs the detected defect category.
[0009] The trained convolutional neural network model was used as the surface defect detection model.
[0010] As a further preferred embodiment, the feature extractor and classifier adopt an independent training strategy, that is, the feature extractor is first trained based on the tactile signal of the product surface and its defect category label; then, the representation vector is obtained through the trained feature extractor, and the classifier is trained based on the representation vector and the corresponding defect category label.
[0011] As a further preferred option, the loss function used during feature extractor training is as follows:
[0012]
[0013] in, Let K represent the loss function of the feature extractor, and let n represent the number of classes. k This represents the number of samples in each category. This represents the set of positive samples of the same class as sample i. Let represent the representation vectors corresponding to samples i, p, and a, respectively; τ represent the hyperparameter controlling the similarity between representation vectors; and W represent the learnable weight matrix. This represents the regularization penalty term based on the Gram matrix.
[0014] As a further preferred option, during feature extraction training, the SGD optimizer is used to optimize the parameters of each layer of the feature extractor generation by generation.
[0015] As a further preferred option, the cross-entropy loss function is used during classifier training, and the Adam optimizer is used to optimize the parameters of each layer of the classifier generation by generation.
[0016] As a further preferred embodiment, the feature extractor uses ResNet18 as its skeleton and employs depth-wise convolution and point-wise convolution as convolutional layers, with the GeLU activation function applied after each convolutional layer to achieve a non-linear transformation.
[0017] As a further preferred embodiment, the classifier adopts an FFN architecture, which includes an input layer, a hidden layer, and an output layer, wherein the hidden layer uses Dropout randomly deactivated neurons.
[0018] As a further preferred embodiment, the method for obtaining the tactile signals of the product surface and their defect category labels includes:
[0019] Based on the 3D model of the product, plan the robotic arm's operating trajectory;
[0020] The robotic arm moves to allow the tactile sensor fixed at its end to touch all the surface areas of the product to be tested, and collect tactile signals.
[0021] Tactile signals are manually categorized to form defect category labels.
[0022] According to a second aspect of the present invention, a surface defect detection model is provided, which is constructed using the above-described method for constructing a surface defect detection model based on tactile feedback.
[0023] According to a third aspect of the present invention, a surface defect detection method is provided, comprising the following steps: inputting a tactile signal of the surface of a product to be tested into the surface defect detection model described above; a feature extractor outputting a representation vector based on the tactile signal; and a classifier outputting the detected defect category based on the representation vector.
[0024] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:
[0025] 1. This invention constructs a defect detection model based on tactile signals, enabling it to capture morphological features at the 10-micrometer level, thereby improving the ability to perceive minute surface defects and achieving high-precision automated detection of minute surface defects in products.
[0026] 2. This invention achieves precise capture of surface defects in transparent, highly reflective, and complex-textured materials through tactile signals, overcoming the limitations of visual methods on such materials; moreover, no light source is required during the tactile signal acquisition process. Therefore, compared to computer vision-based methods, the method of this invention has a wider range of applications.
[0027] 3. This invention adopts a class-guided contrastive learning strategy, which utilizes prior label knowledge to optimize the training process of contrastive learning. It extends the micro-level contrastive constraints at the sample level to the macro-level contrastive constraints at the class level, realizing the compressed distribution of samples of the same class and the differential representation of samples of different classes, which significantly improves the detection accuracy and precision of the model.
[0028] 4. The independent training of the feature extractor and classifier adopted in this invention provides a simpler way to fine-tune the model, ensuring that the model training achieves optimal performance. Attached Figure Description
[0029] Figure 1 This is a flowchart of a product surface micro-defect detection method based on machine tactile sensing, according to an embodiment of the present invention.
[0030] Figure 2 A schematic diagram of a tactile experimental platform constructed according to an embodiment of the present invention;
[0031] Figure 3 This is a schematic diagram showing the tactile signals and dimensions of different types of surface micro-defects collected in an embodiment of the present invention.
[0032] Figure 4 A structural diagram of the surface defect detection model constructed in an embodiment of the present invention;
[0033] Figure 5 This is a flowchart illustrating the training principle of the defect detection model in an embodiment of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0035] This invention provides a method for constructing a surface defect detection model based on tactile feedback, such as... Figure 1 As shown, it includes the following steps:
[0036] S1. Obtain tactile signals from the product surface and label them with defect category tags to construct a tactile signal dataset.
[0037] Specifically, tactile signals from the product surface are acquired through tactile sensors. In this embodiment, a tactile testing platform is constructed, which includes a robotic arm, a gripping device, tactile sensors, and an engineering host, such as... Figure 2 As shown in the diagram. The main control unit is used for data analysis and controlling the movement of the robotic arm, while the gripping device is used to fix the tactile sensor, which is used to acquire tactile signals.
[0038] Based on the 3D model of the workpiece to be tested, the trajectory of the robotic arm is planned to ensure that the tactile sensor can contact all the areas to be tested on the workpiece, collecting tactile signals as the medium for detecting minute surface defects. Specifically, path instructions are written using the Python SDK and imported into the host computer. The host computer sends instructions to control the robotic arm to move and hold the tactile sensor, bringing the sensor's measuring end into contact with the product to obtain the tactile signal. The resolution is 64×64, and the signal is stored as a .png file. The tactile signal is manually labeled to indicate what type of surface defect it represents. Some defect categories of tactile signals are as follows: Figure 3 As shown. Based on this, a tactile signal dataset is constructed.
[0039] It should be noted that this is only a preferred embodiment, and other robots or manual methods can also be used to obtain tactile signals; that is, the specific method and structure are not limited, as long as the surface tactile signals can be obtained through a tactile sensor.
[0040] S2. Construct a convolutional neural network model for tactile signal analysis.
[0041] The TouchNet convolutional neural network model for haptic signal analysis was built using the PyTorch library. This model includes a feature extractor and a classifier, such as... Figure 4 As shown. The feature extractor extracts data from the raw tactile signals. Extract discriminative features that can distinguish signal categories and compress them into representation vectors. Classifier for decoding Information and mapping it to classification probabilities
[0042] Furthermore, the feature extractor uses ResNet18 as its skeleton and embeds depth-wise and point-wise convolutions to compress model parameters. (Representation vector) The dimension is 512. Each convolutional layer is followed by a GeLU activation function to achieve a non-linear transformation.
[0043] Furthermore, the classifier uses an FFN architecture, consisting of three neural network layers: an input layer, a hidden layer, and an output layer. The hidden layer has 64 neurons, and Dropout is used to randomly deactivate neurons to avoid overfitting.
[0044] S3, using tactile signal datasets Training the TouchNet model, such as Figure 5 As shown, the training process optimizes the model parameters generation by generation, enabling it to effectively detect and classify different types of defects, thus obtaining a surface defect detection model.
[0045] Specifically, the feature extractor and classifier employ independent training strategies, including:
[0046] (1) The input to the feature extractor is tactile signals. and its category tags The output is a vector. By training the feature extractor using a class-guided contrastive learning strategy, and utilizing prior knowledge of class labels, the distribution space of signals of the same class is compressed, while the spatial distance of signals of different classes is stretched, thereby guiding the hyperspherical clustering process during training at a macroscopic class level.
[0047] For a K-class classification task, suppose each class has n k For sample x, then i Positive sample set negative sample set That is, sample x i Each sample in a class is a positive sample pair with the remaining samples in that class, and a negative sample pair with samples from other classes. Therefore, the loss function for class-guided contrastive learning is:
[0048]
[0049] Where W is the learnable weight matrix, K is the total number of classes, k is the sample set of a certain class, and n k n is the number of samples in the k-sample set. s It is the total number of samples from all classes except class k, where i, j, and k are distinct samples; a second regularization is applied to the weight vector. Penalizing similarity and promoting orthogonality of W optimizes the training process and avoids feature decay. The loss function of the feature extractor is then... for:
[0050]
[0051] in This represents the set of positive samples of the same class as sample i, and temperature τ is the control d. rep The hyperparameters for similarity between features are determined. The training process utilizes the SGD optimizer to progressively optimize the parameters of each layer of the feature extractor.
[0052] (2) The input to the classifier is a representation vector. and defect category labels The output is the detected defect category. After the feature extractor is trained, the classifier is trained using cross-entropy loss. For a K-class classification task, assuming each class has n... k For each sample, the loss function of the classifier is... for:
[0053]
[0054] in, For the true category of defects, This is the detection result output by the classifier. The training process uses the Adam optimizer to optimize the parameters of each layer of the classifier generation by generation.
[0055] Furthermore, the surface defect detection model constructed based on the above method achieves micron-level surface defect detection, including the following steps:
[0056] S4. Obtain the tactile signal of the product surface under test and input it into the surface defect detection model to obtain the corresponding defect category.
[0057] Specifically, a pre-trained surface defect detection model is deployed to the engineering host of the tactile testing platform using Docker for analyzing the tactile signals of the products to be inspected. The tactile testing platform then collects the tactile signals from the products and inputs them into the surface defect detection model on the engineering host. A feature extractor extracts the representation vectors of the tactile signals, and a classifier classifies the feature vectors to obtain the detection results for the samples. In the inspection process, the engineering host sends control commands to the robotic arm, causing it to move to the target position and bring the measuring end of the tactile sensor into contact with the target location. The signals collected by the tactile sensor are transmitted to the engineering host in real time and input into the pre-deployed defect detection model for processing. The defect detection model performs online analysis of the received tactile signals and outputs the defect detection results at the target location in real time.
[0058] The following are specific examples:
[0059] To verify the practical application effect of the method of the present invention, tactile signals of common surface defects such as normal surfaces, bumps, pits, and scratches were collected using a constructed tactile experimental platform. A tactile signal dataset was built and used to verify the algorithm. In this embodiment, the robotic arm used is JAKA Zu5, the tactile sensor is Gelsight mini, and the industrial host is configured with a Core i5 14600kf CPU and an Nvidia RTX 4070Ti Super GPU.
[0060] VGG16, ResNet18, ResNet50, DEGAN, and MoCo were selected as comparison models for TouchNet. The complexity, computational efficiency, and accuracy of each model were comprehensively evaluated by calculating multiple metrics, including the number of parameters, MACs, sample computation time, and accuracy. Each experiment underwent five independent training runs. The trained models were then deployed on a haptic testing platform, and their performance was tested using untrained, novel test pieces. The average results of the five experiments were used as the final evaluation metric, as shown in Table 1.
[0061] Table 1 Comparison of experimental results
[0062]
[0063] As clearly shown in Table 1, the TouchNet model proposed in this invention achieves better overall performance. It is worth noting that the MoCo model, based on the classic contrastive learning framework, performs poorly, mainly due to the inherent limitations of its unsupervised training method. In contrast, the TouchNet model based on class-guided contrastive learning proposed in this invention significantly improves accuracy and greatly reduces the number of parameters with a slight increase in computation time. This experimental result verifies that the introduction of prior labels can effectively guide high-dimensional feature clustering and improve the model's ability to mine discriminative features. On the other hand, the depth-wise and point-wise convolutions in the TouchNet feature extractor are hardware-friendly operations that can significantly compress the number of parameters, but inevitably increase the computational cost and time slightly. Furthermore, although the DEGAN model based on the generative framework also achieves high accuracy, its complex structure results in a much higher number of parameters and computational cost than other models, thus making it susceptible to limitations in computational resources in practical applications.
[0064] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for constructing a surface defect detection model based on tactile feedback, characterized in that, Includes the following steps: The convolutional neural network model is trained using tactile signals from the product surface and their defect category labels. This model includes a feature extractor and a classifier. During training: The feature extractor takes the tactile signal of the product surface and its defect category label as input. It adopts a class-guided contrastive learning strategy to extract discriminative features that can distinguish the signal category from the tactile signal and compress them into a representation vector as output. The classifier takes the representation vector and the corresponding defect category label as input and outputs the detected defect category. The feature extractor and classifier employ an independent training strategy. First, the feature extractor is trained based on the tactile signals of the product surface and its defect category labels. Then, the representation vector is obtained through the trained feature extractor, and the classifier is trained based on the representation vector and the corresponding defect category labels. Specifically, the input to the feature extractor is the tactile signal from the product surface. and its defect category labels The output is a vector. The feature extractor is trained by a class-guided contrastive learning strategy. By utilizing prior knowledge of class labels, the distribution space of signals of the same class is compressed and the spatial distance of signals of different classes is stretched, thereby guiding the hyperspherical clustering process during training at the macro-level of categories. For a K-class classification task, suppose each class has For each sample, then for the sample Positive sample set negative sample set That is, sample If a sample is paired with the remaining samples of the same class as a positive sample pair, and with samples from other classes as a negative sample pair, then the loss function for class-guided contrastive learning is... for: Where W is the learnable weight matrix, K is the total number of classes, and k is the sample set of a certain class. It is the number of samples in the k-sample set. It is the total number of samples from all classes except class k. , , These are different samples; apply quadratic regularization to the weight vector. Punish similarity and promote W Orthogonalization optimizes the training process and avoids feature decay; therefore, the loss function used during feature extractor training is as follows: in, This represents the loss function of the feature extractor. Representation and Sample i A set of positive samples of the same type, , , Representing samples respectively i , p , a The corresponding representation vector, This represents a hyperparameter that controls the similarity between representation vectors. This represents the regularization penalty term based on the Gram matrix; The trained convolutional neural network model was used as the surface defect detection model.
2. The method for constructing a surface defect detection model based on tactile feedback as described in claim 1, characterized in that, During feature extractor training, the SGD optimizer is used to optimize the parameters of each layer of the feature extractor generation by generation.
3. The method for constructing a surface defect detection model based on tactile feedback as described in claim 1, characterized in that, The classifier is trained using the cross-entropy loss function, and the Adam optimizer is used to optimize the parameters of each layer of the classifier generation by generation.
4. The method for constructing a surface defect detection model based on tactile feedback as described in claim 1, characterized in that, The feature extractor uses ResNet18 as its skeleton and employs depth-wise and point-wise convolutions as convolutional layers. After each convolutional layer, a GeLU activation function is used to achieve a non-linear transformation.
5. The method for constructing a surface defect detection model based on tactile feedback as described in claim 1, characterized in that, The classifier adopts an FFN architecture, which includes an input layer, a hidden layer, and an output layer. The hidden layer uses Dropout random deactivated neurons.
6. The method for constructing a surface defect detection model based on tactile feedback as described in any one of claims 1-5, characterized in that, The method for obtaining the tactile signals of the product surface and their defect category labels includes: Based on the 3D model of the product, plan the robotic arm's operating trajectory; The robotic arm moves to allow the tactile sensor fixed at its end to touch all the surface areas of the product to be tested, and collect tactile signals. Tactile signals are manually categorized to form defect category labels.
7. A surface defect detection model, characterized in that, It is constructed using the surface defect detection model construction method based on tactile feedback as described in any one of claims 1-6.
8. A method for detecting surface defects, characterized in that, The process includes the following steps: inputting the tactile signal of the product to be tested into the surface defect detection model as described in claim 7; the feature extractor outputs a representation vector based on the tactile signal; and the classifier outputs the detected defect category based on the representation vector.