A tire bulge defect detection method based on multi-modal information fusion

Abstract

This invention proposes a tire bulge defect detection method based on multimodal information fusion. The multimodality is manifested in the fusion of two-dimensional and three-dimensional information. The two-dimensional information is a two-dimensional planar image captured by an industrial camera; the three-dimensional information is tire point cloud data obtained through a laser 3D scanner. In feature processing, a feature extraction algorithm is designed to enhance image features, obtaining a high-precision pseudo-color image that can better characterize bulge-type defects. After obtaining the high-precision pseudo-color image, it is fused with the corresponding two-dimensional planar image along the channel direction to obtain multimodal features, and ResNet34 is selected as the defect classifier. This invention, based on deep learning methods, fully integrates two-dimensional and three-dimensional feature information to obtain multimodal features, effectively identifying tire bulge defects, avoiding interference from external factors such as lighting and viewing angle on defect sorting, thus helping to improve screening efficiency and ensure product quality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and deep learning, and is a method for detecting tire bulge defects based on multimodal information fusion. Background Technology

[0002] As living standards improve, cars have become the preferred mode of transportation for short-distance travel, and tire quality is closely related to everyone's safety. Due to the complexity of the tire manufacturing process and the many factors affecting product quality, defects cannot be completely avoided during manufacturing. Therefore, tires undergo careful quality inspection before leaving the factory to prevent defective products from entering the market and becoming safety hazards.

[0003] Currently, tire appearance quality inspection relies primarily on manual methods. This involves workers observing each tire individually and manually inspecting its surface. This method is highly dependent on the inspector's experience, focus, and skill level. Furthermore, variations in ambient lighting conditions, worker fatigue levels, and differing evaluation standards can lead to missed defects, subjectivity, and low efficiency. Due to these drawbacks, automated inspection technology is urgently needed for tire defect detection. However, for some defects, such as sidewall blisters, two-dimensional images struggle to provide depth information and fail to effectively utilize height features, hindering effective detection.

[0004] With the development of technology, 3D vision technology has gradually matured. 3D point cloud data can accurately describe complex models through precise and dense point mapping, obtaining the 3D coordinate information of objects. Therefore, this invention utilizes a laser 3D scanner to acquire 3D point cloud data of tires, while simultaneously using an industrial camera to acquire 2D planar images of the tires. By fully integrating 2D and 3D information, this invention effectively improves tire bulge defect detection technology, overcomes the limitations of traditional 2D image defect detection methods, and promotes the automation and intelligent development of tire quality inspection technology. Summary of the Invention

[0005] The purpose of this invention is to provide a tire bulge defect detection method based on multimodal information fusion, addressing the shortcomings of existing technologies.

[0006] Therefore, the present invention adopts the following technical solution:

[0007] A tire bulge defect detection method based on multimodal information fusion is characterized by: the method fusing features of two-dimensional planar images and three-dimensional point cloud data; by processing the three-dimensional point cloud data of the tire to generate a height feature pseudo-color image for feature extraction, and simultaneously extracting features from the two-dimensional planar image; then, the two types of features are stitched together on the channel to obtain fused features for training the tire bulge defect detection model; the tire bulge defect detection method includes model training and defect detection.

[0008] The model training includes the following steps:

[0009] Step S1, Data Acquisition: The tire is scanned using a laser 3D scanner and an industrial camera to obtain the original 3D point cloud data file of the tire in CSV format and the 2D planar image of the tire in BMP format. Information is extracted from the 3D point cloud data, and the 3D point cloud coordinates are extracted and stored in a TXT format file as data for subsequent steps.

[0010] Step S2, Data Preprocessing: Further processing is performed on the txt format file generated in step S1; Since the tire hair on the tire surface and the noise present during the scanning process can interfere with the result judgment, it is necessary to perform radius filtering on the three-dimensional point cloud to remove the influence of the noise on defect classification;

[0011] Step S3, Generation of height feature pseudo-color image: In order to make full use of the depth information of the 3D point cloud data and overcome the shortcomings of 2D images in representing bulge-like defects, this invention uses a height feature pseudo-color image that reflects the characteristics of bulge-like defects as a point cloud feature; the specific method for generating the height feature pseudo-color image includes the following steps:

[0012] Step Z1, Point Cloud Sampling: Since the height between adjacent points is almost the same, the 3D point cloud is uniformly sampled in order to reduce the computational cost.

[0013] Step Z2, relative height calculation: Traverse the sampled point cloud 3D coordinates, obtain the minimum and maximum values ​​in the z-direction, use the difference between the two as the subsequent normalization value, and use the minimum value in the z-direction as the reference value. Then traverse the sampled point cloud 3D coordinates again, calculate the difference between the z-direction coordinate of each point and the reference value, divide by the normalization value to obtain the relative height value and save it;

[0014] Step Z3, Generate relative height grayscale image: Create a black and white image of the same size as the sampled image, set the background color of the image to black by default; traverse the sampled point cloud, multiply the relative height value of the point cloud at the corresponding position in the black and white image by 255 to obtain the grayscale value of that pixel, and obtain the relative height grayscale image after completing the traversal.

[0015] Step Z4, Generate a high-feature pseudo-color image: Since the height difference is small, the grayscale image is difficult to represent the defect. Therefore, the image features are enhanced by the LUT (Look Up Table) algorithm. The LUT algorithm outputs the grayscale value as another set of RGB values, thereby changing the exposure and color of the image, highlighting the defect features, and generating a high-feature pseudo-color image.

[0016] Step S4, Multimodal Feature Fusion: The height feature pseudo-color image obtained in Step S3 is input into a three-layer convolutional neural network with a kernel size of 3×3, and the number of channels are 16, 32 and 1 respectively, with a stride of 1 for each layer; the resulting feature map is input into a softmax layer to obtain a height attention feature map; the obtained height attention feature map is multiplied by a two-dimensional planar image to obtain a feature-enhanced two-dimensional image, which is used to compensate for the shortcomings of two-dimensional planar images in representing bulge features; the feature-enhanced two-dimensional image and the height feature pseudo-color image are concatenated along the channel dimension to obtain the fused feature.

[0017] Step S5, Defect Detection: Input the multimodal feature map obtained in step S4 into the ResNet34 defect classifier to obtain the defect classification result; calculate the cross-entropy loss between the classification result and the real sample label as the classification loss of the network;

[0018] Step S6, Update parameters: Repeat step S5 to update the model parameters using stochastic gradient descent until the maximum number of iterations is reached;

[0019] Step S7, Save Model: Save the defect detection model from step S6;

[0020] The defect detection includes the following steps:

[0021] Step T1, Data Acquisition: The tire to be inspected is scanned using a laser 3D scanner and an industrial camera to acquire the 3D point cloud data of the tire outline and the 2D planar image of the tire; then, the obtained 3D point cloud data is subjected to radius filtering to remove tire hairs and noise present during the tire scanning process.

[0022] Step T2, Feature Extraction: The point cloud data obtained in step T1 is subjected to feature extraction according to step S3 to obtain a height feature pseudo-color image;

[0023] Step T3, Multimodal Feature Fusion: The height feature pseudo-color image obtained in step T2 is fused with the tire 2D planar image using multimodal features;

[0024] Step T4, Defect Detection: Input the multimodal features obtained in step T3 into the ResNet34 classifier saved in step S7 for defect classification, and finally obtain the defect detection result.

[0025] According to the purpose of this invention, a tire bulge defect detection method based on multimodal information fusion is proposed, characterized by training using the above-mentioned model training method.

[0026] According to the purpose of this invention, a point cloud feature extraction method is proposed. The method is characterized by using radius filtering to remove noise from a three-dimensional point cloud image, uniformly sampling the point cloud, calculating the relative height to obtain a relative height grayscale image, using the LUT algorithm to optimize the relative height grayscale image, and finally obtaining a height feature pseudo-color image.

[0027] The tire bulge defect detection method based on multimodal information fusion provided by this invention has the following advantages:

[0028] (1) Generate a dataset by generating a pseudo-color image of height features based on the three-dimensional point cloud data of the tire. Utilize the characteristic that the height of tire bulge defects is different from that of normal surfaces to mine the depth information of the three-dimensional point cloud data and realize the detection of bulge defects.

[0029] (2) Generate a relative height grayscale image based on 3D point cloud data, and convert the grayscale value into the corresponding RGB value through LUT algorithm to enhance the defect features;

[0030] (3) By integrating the features of two-dimensional planar images and the pseudo-color images with height features, a method for detecting tire bulge defects based on multimodal information fusion is proposed. Attached Figure Description

[0031] Figure 1 This is an overall flowchart of the method of the present invention;

[0032] Figure 2 This is a flowchart of the training process for the bulge defect detection model of the present invention;

[0033] Figure 3 This is a partial crop of the relative height grayscale image and the height feature pseudocolor image proposed by the method of the present invention;

[0034] Figure 4 This is a schematic diagram of the feature fusion process of the method of the present invention;

[0035] Figure 5 This is a flowchart of the main process of the defect detection stage of the method of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be described in detail with reference to the accompanying drawings and specific implementation examples of tire defect detection for bubbling-type defects.

[0037] Specifically, the detection of bulging defects is carried out, and the overall process is as follows: Figure 1As shown; data acquisition is performed using a laser contour sensor, and feature extraction is performed on the obtained point cloud data to obtain a height feature pseudo-color image. The feature image is then cropped to a size of 256x256 using a sliding window method as the dataset. The dataset used for model training consists of 1420 images of bulge-type defects and 2560 images of flawless images. A model for tire defect classification is obtained through model training, and then the model is validated. The flowchart of the bulge defect detection model training is shown below. Figure 2 As shown, the specific steps are as follows:

[0038] Step S1, Data Acquisition: A laser contour sensor and an industrial camera are used to scan the tire to acquire 3D tire data and 2D planar images, and to mark the locations of defects. The point cloud data directly acquired by the sensor is in CSV format, and in addition to the 3D coordinates of the contour points, it also contains the sensor's built-in parameter information; therefore, it needs to be converted into a specified TXT format file using a computer program to prepare for subsequent data preprocessing.

[0039] Step S2, Data Preprocessing: Further processing is performed on the txt format file mentioned in Step S1. Since tire hairs on the tire surface and noise present during the scanning process can interfere with the result judgment, a radius filtering algorithm is used to filter the 3D point cloud to remove the influence of the aforementioned noise on defect classification.

[0040] Step S3, Generation of height feature pseudo-color image: In order to make full use of the depth information of 3D point cloud data, this invention uses a height feature pseudo-color image as the feature image; the specific generation method is as follows:

[0041] Step Z1, Point Cloud Sampling: Since the heights of adjacent points in the point cloud are almost the same, in order to reduce computational costs, the 3D point cloud is uniformly sampled with a sampling step size of 0.2 in the x-direction and 1.1 in the y-direction.

[0042] Step Z2, relative height calculation: First, traverse the sampled point cloud array, take the minimum value in the z direction as the reference value, and take the difference between the maximum value in the z direction and the minimum value in the z direction as the normalization value; then traverse the sampled point cloud array again, calculate the difference between the z direction coordinate of each group of point clouds and the reference value, divide by the normalization value to obtain the relative height value and save it.

[0043] Step Z3, Generate Relative Height Grayscale Image: Create a black and white image. The image length is the difference between the maximum and minimum values ​​in the x-direction divided by 0.2; the image width is the difference between the maximum and minimum values ​​in the y-direction divided by 1.1; set the image background color, which is black by default; traverse the sampled point cloud, multiply the relative height value of the point cloud at the corresponding position in the black and white image by 255 to obtain the grayscale value of that pixel. After completing the traversal, the relative height grayscale image will be obtained; the formula is expressed as:

[0044]

[0045] Among them, h x,y z represents the relative height of the point with coordinates (x, y). x,y Let z represent the z-value of the point (x, y). min For the minimum z value, z max The maximum z value;

[0046] Step Z4, generating a height feature pseudo-color image: Due to the small height difference, grayscale images are insufficient to represent defects. Therefore, the image features are enhanced using the LUT (Look Up Table) algorithm. The LUT algorithm outputs grayscale values ​​as another set of RGB values, mapping them to a color image to highlight defect features. The height feature pseudo-color image generated from the relative height grayscale image after optimization by the LUT algorithm is shown below. Figure 3 As shown;

[0047] Step S4, Dataset Creation: The height feature pseudo-color image obtained in Step S3 and its corresponding tire 2D plane image are segmented using a sliding window. An IoU function is defined to find the corresponding location of the defect in the height feature pseudo-color image based on the defect location marked in the tire 2D plane image. If the coverage of the sliding window and the marked area reaches more than 80%, it is considered a defective sample. The window image is cropped, and a tire 2D plane image of the corresponding location and size is found proportionally and saved to the dataset labeled as defective. Otherwise, it is considered a flawless sample. To maintain a balance between positive and negative samples, a random number judgment condition is set before saving flawless images. An integer between 1 and 10 is randomly generated. When the random number is less than the set threshold of 7, the current image is saved to the flawless dataset. While saving the height feature pseudo-color image dataset, the corresponding location is found in the 2D plane image and cropped to create a corresponding 2D plane image dataset. The dataset created according to Step S4 consists of 1420 defective images (bulge type) and 2560 flawless images.

[0048] Step S5, Multimodal Feature Fusion: The labeled training set obtained in Step S4 is subjected to feature extraction and fusion. The height feature pseudo-color image obtained in Step S4 is input into a three-layer convolutional neural network, with kernel sizes of 3×3, channel numbers of 16, 32, and 1 respectively, and a stride of 1 for each layer. The resulting feature map is then fed into a Softmax layer to obtain a height attention feature map. Next, the obtained height attention feature map is multiplied by a two-dimensional planar image to obtain a feature-enhanced two-dimensional image, which compensates for the difficulty of two-dimensional planar images in representing bulge features. Finally, the feature-enhanced two-dimensional image and the height feature pseudo-color image are concatenated along the channel dimension to obtain the fused feature. The feature fusion process is as follows: Figure 4 As shown;

[0049] Step S6, Defect Classification: Input the multimodal feature map obtained in step S5 into the ResNet34 classifier to obtain the defect classification result. Calculate the cross-entropy loss between the classification result and the real sample label as the classification loss of the network.

[0050] Step S7, update parameters: Repeat step S6 to update the model parameters using stochastic gradient descent until the maximum number of iterations is reached; the initial learning rate is 0.001, and the total number of iterations is 1500.

[0051] Step S8, Save the model: Save the defect classification model parameters from step S6;

[0052] The defect detection process is as follows: Figure 5 As shown, defect detection includes the following steps:

[0053] Step T1, Data Acquisition: Use a laser 3D scanner to scan the tire that needs quality inspection to obtain a 3D point cloud image of the tire outline and a 2D planar image of the tire; then perform radius filtering on the obtained 3D point cloud data to remove tire hairs and noise present during the tire scanning process.

[0054] Step T2, Feature Extraction: The point cloud data obtained in step T1 is subjected to feature extraction according to step S3 to obtain a height feature pseudo-color image;

[0055] Step T3, Feature Fusion: Perform multimodal feature fusion between the height feature pseudo-color image obtained in step T2 and the two-dimensional planar image of the tire;

[0056] Step T4, Defect Detection: Input the multimodal features obtained in step T3 into the ResNet34 classifier saved in step S7 for defect classification, and finally obtain the defect detection result;

[0057] Results Verification: To verify the effectiveness of the tire bulge defect detection method based on multimodal information fusion proposed in this invention, this method was applied in a practical application and achieved a good accuracy of 91.32%. The method proposed in this invention fully utilizes the depth information of point clouds and simultaneously fuses two-dimensional image information to obtain multimodal features, applying multimodal fusion detection technology to the field of tire defect detection.

[0058] The technical solutions of the present invention have been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for detecting tire bulge defects based on multimodal information fusion, characterized in that: The tire bulge defect detection method is based on two-dimensional planar images and three-dimensional point cloud data. It utilizes the feature fusion of two-dimensional and three-dimensional information to achieve tire bulge defect detection. The tire bulge defect detection method includes defect detection model training and defect detection. The defect detection model training is performed according to the following steps: Step S1, Data Acquisition: The tire is scanned using a laser 3D scanner and an industrial camera to obtain the original 3D point cloud data file of the tire in CSV format and the 2D planar image of the tire in BMP format. Information is extracted from the 3D point cloud data, and the 3D point cloud coordinates are extracted and stored in a TXT format file as data for subsequent steps. Step S2, Data Preprocessing: Further processing is performed on the txt format file generated in step S1; Since the tire hair on the tire surface and the noise present during the scanning process can interfere with the result judgment, it is necessary to perform radius filtering on the three-dimensional point cloud to remove the influence of the noise on defect classification; Step S3, generation of height feature pseudo-color image: In order to make full use of the depth information of the three-dimensional point cloud data and overcome the shortcomings of two-dimensional images in representing bulge-like defects, the height feature pseudo-color image that reflects the characteristics of bulge-like defects is used as the point cloud feature. The specific method for generating a high-resolution pseudo-color image includes the following steps: Step Z1, Point Cloud Sampling: Since the heights of adjacent points in the point cloud are almost the same, the 3D point cloud is uniformly sampled in order to reduce computational costs. Step Z2, Height Calculation: Traverse the three-dimensional coordinates of the sampled point cloud, obtain the minimum and maximum values ​​in the z-direction, use the difference between the two as the subsequent normalization value, and use the minimum value in the z-direction as the reference value; then traverse the three-dimensional coordinates of the sampled point cloud again, calculate the difference between the z-direction coordinate of each point and the reference value, divide by the normalization value to obtain the height value and save it. Step Z3, Generate a height grayscale image: Create a black and white image of the same size as the sampled image, set the background color of the image to black by default; traverse the sampled point cloud, multiply the height value of the point cloud at the corresponding position in the black and white image by 255 to obtain the grayscale value of the point cloud, and obtain the height grayscale image after completing the traversal. Step Z4, Generate a pseudo-color image with height features: Since the height difference is small, grayscale images are difficult to represent defects. Therefore, the image features are enhanced by the LUT (Look Up Table) algorithm. The LUT algorithm outputs grayscale values ​​as another set of RGB values, thereby changing the exposure and color of the image and highlighting the flaws. Step S4, Multimodal Feature Fusion: The high-modality pseudo-color image obtained in Step S3, which can well represent bulge-like defects, is input into a three-layer convolutional neural network. The kernel size of each layer is 3×3, the number of channels is 16, 32 and 1 respectively, and the stride is 1. The resulting feature map is input into a softmax layer to obtain a high-attention feature map. The high-attention feature map is multiplied by the two-dimensional planar image to obtain a feature-enhanced two-dimensional image, which is used to make up for the shortcomings of the two-dimensional planar image in that it is difficult to represent bulge features. Finally, the feature-enhanced 2D image and the high-precision pseudo-color image are stitched and fused along the channel dimension to obtain a multimodal feature map; Step S5, Defect Detection: Input the multimodal feature map obtained in step S4 into the ResNet34 classifier to obtain the defect classification result; calculate the cross-entropy loss between the classification result and the real sample label as the classification loss of the network; Step S6, Update parameters: Repeat step S5 to update the model parameters using stochastic gradient descent until the maximum number of iterations is reached; Step S7, Save Model: Save the defect detection model from step S6; The defect detection includes the following steps: Step T1, Data Acquisition: The tire to be inspected is scanned using a laser 3D scanner and an industrial camera to acquire the 3D point cloud data of the tire outline and the 2D planar image of the tire; then, the obtained 3D point cloud data is subjected to radius filtering to remove tire hairs and noise present during the tire scanning process. Step T2, Feature Extraction: The point cloud data obtained in step T1 is subjected to feature extraction according to step S3 to obtain a height feature pseudo-color image; Step T3, Multimodal Feature Fusion: The height feature pseudo-color image obtained in step T2 is fused with the tire 2D planar image using multimodal features; Step T4, Defect Detection: Input the multimodal features obtained in step T3 into the ResNet34 classifier saved in step S7 for defect classification, and finally obtain the defect detection result.

2. The tire bulge defect detection method based on multimodal information fusion according to claim 1, characterized in that, A dataset was created by segmenting images using a sliding window, which was then used to train a tire bulge defect detection model.

3. The tire bulge defect detection method based on multimodal information fusion according to claim 1, characterized in that, In step S5, ResNet32 is used as the defect classifier, and pre-trained weights on ImageNet are used as initialization parameters.

Images