Industrial part defect detection method, system and device based on 2D-3D multi-modal images and storage medium

Abstract

The application relates to an industrial part defect detection method, system, device and storage medium based on 2D-3D multi-modal images. The industrial part defect detection method comprises the following steps: firstly, acquiring 2D images and 3D point cloud data of a target part by using an image acquisition device; then, pre-processing the 2D images and 3D point cloud by using a data processing module, detecting and classifying defects by using a deep learning model, fusing and comprehensively evaluating 2D and 3D features by using a result fusion module, and obtaining a final defect detection result. Compared with a traditional single modal detection method, the multi-modal system has the following advantages: firstly, 2D and 3D information is comprehensively utilized, and the accuracy and robustness of detection are enhanced; secondly, the multi-modal system is suitable for different types of industrial parts and has good universality; and thirdly, the application of the deep learning model reduces the demand for artificial feature engineering and improves the automation degree of detection.

Description

TECHNICAL FIELD

[0001] The present application relates to a defect detection technology, in particular to an industrial part defect detection method, system, device and storage medium based on 2D-3D multi-modal images. BACKGROUND

[0002] In the industrial manufacturing process, defect detection of parts is one of the key steps to ensure product quality. However, traditional single modal defect detection methods (such as using only 2D images or only using 3D point cloud data) may have limitations when facing complex part surface defects, and cannot accurately detect and classify defects. Therefore, a more comprehensive and accurate defect detection method is needed, which can fully utilize the information of multiple data sources.

[0003] 2D images have advantages in surface defect detection, which can capture the texture, color and shape of the part features. However, 2D images are affected by factors such as lighting, occlusion and viewing angle, which may lead to a decrease in detection accuracy. On the other hand, 3D point cloud data can provide geometric information of the part, but in some cases there may be noise and incompleteness, affecting the reliability of defect detection.

[0004] In order to overcome these problems, the present application proposes an industrial part defect detection method and system based on 2D-3D multi-modal images. By simultaneously using 2D images and 3D point cloud data, their advantages in surface features and geometric information can be fully utilized, thereby improving the accuracy and robustness of defect detection. Reduce the need for manual feature engineering, and improve the degree of automation of detection. SUMMARY

[0005] For the problem of industrial part defect detection, the first aspect of the present application provides an industrial part defect detection method based on 2D-3D multi-modal images, which comprises:

[0006] Using an industrial camera to collect 2D and 3D images of the part to be detected, by comprehensively utilizing 2D and 3D image data, using a deep learning model for feature extraction, and through feature fusion to realize high-precision and general-purpose defect detection, the industrial part defect detection method comprises the following steps:

[0007] Obtain 2D image data of the target industrial part from the image acquisition device; obtain depth image and corresponding 3D point cloud data of the target industrial part from the image acquisition device;

[0008] Perform image enhancement processing on the obtained 2D image data to improve image quality, and perform center random cropping and image foreground extraction processing to ensure the training effect of the detection model; and perform filtering and denoising processing on the obtained 3D point cloud data to reduce data noise;

[0009] extracting features from the pre-processed 2D image data based on a convolutional neural network to generate a 2D feature vector; and extracting features from the pre-processed 3D point cloud data using a point cloud processing algorithm to generate a 3D feature vector;

[0010] performing feature fusion on the 2D feature vector and the 3D feature vector to obtain a 2D-3D multi-modal feature;

[0011] designing a discriminator model using deep learning technology, performing defect detection and classification based on the 2D-3D multi-modal feature, and determining a possible defect point position according to a comprehensive defect evaluation result.

[0012] In some embodiments, the extracting features from the pre-processed 2D image data based on a convolutional neural network to generate a 2D feature vector includes: using a ResNet backbone network pre-trained for image classification to extract high-level abstract features from the pre-processed 2D image, saving two level feature maps of different sizes, performing linear mapping and feature splicing to obtain a final 2D feature map.

[0013] In some embodiments, the extracting features from the pre-processed 3D point cloud data using a point cloud processing algorithm to generate a 3D feature vector includes: grouping points using a FPS farthest point sampling technique, and using a PointFormer network to perform point cloud feature extraction to obtain a 3D feature vector for each group.

[0014] In some embodiments, the performing feature fusion on the 2D feature vector and the 3D feature vector includes: using a cross-attention mechanism to fuse the obtained 2D feature and 3D feature to obtain a final positive sample feature real_feat, so that the model can learn the positive sample information.

[0015] In some embodiments, the performing defect detection and classification includes: introducing Gaussian noise to generate different distribution of abnormal samples, i.e., negative sample features fake_feat, while the detection network learns the fused positive samples, and the model treats the positive and negative samples as 0 and 1 labels respectively for training; in the inference detection process, the Gaussian noise is not introduced, but the abnormal pictures are directly detected, and according to the defect evaluation result output by the model, a possible defect point position is determined, a heat map is generated to visualize the detection result.

[0016] The second aspect of the present application provides an industrial part defect detection system based on 2D-3D multi-modal images, which comprises:

[0017] An image acquisition device module is configured to acquire 2D image data and 3D point cloud data of the target industrial part;

[0018] A data processing module is configured to perform image enhancement processing on the 2D image data and filtering and denoising processing on the 3D point cloud data;

[0019] A feature extraction module is configured to perform feature extraction on the 2D image data by using a convolutional neural network and perform feature extraction on the 3D point cloud data by using a point cloud processing algorithm;

[0020] A feature fusion module is configured to perform feature fusion on the results obtained after 2D and 3D feature extraction;

[0021] A defect point detection module is configured to perform defect detection and classification and defect point positioning.

[0022] The third aspect of the present application provides an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the industrial part defect detection method of any one of the above.

[0023] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, the computer program is executed by the processor to perform the industrial part defect detection method of any one of the above.

[0024] The beneficial effects of the present application are that the industrial part defect detection method and system based on 2D-3D multi-modal image can comprehensively utilize 2D and 3D image data and combine deep learning model for defect detection, which can more comprehensively evaluate the defect condition of the part compared with the traditional single modal method. It can be applied to different types of industrial parts and has strong universality. Whether it is a planar part, a curved part or a part with complex geometry, it can perform defect detection through multi-modal image data to meet diversified manufacturing needs. It has high automation. Through the deep learning model, the present application reduces the need for manual feature engineering, making the entire defect detection process more automated, reducing the technical requirements of the operator, improving the consistency and repeatability of the detection. It improves production efficiency. Efficient and accurate defect detection helps to reduce the rate of defective products, reduces unnecessary secondary processing and waste, thereby improving production efficiency and reducing costs for enterprises. By outputting the final defect detection result, the operator can quickly locate the defect points on the part and take appropriate repair measures, which helps to solve problems in a timely manner and to a certain extent reduces the delay and loss in production. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 This is a schematic diagram of the industrial parts defect detection system of the present invention;

[0026] Figure 2 This is a flowchart of the feature extraction technology for multimodal images according to the present invention;

[0027] Figure 3 This is a flowchart of the feature fusion technology based on the cross-attention mechanism of the present invention;

[0028] Figure 4 This is a flowchart of the negative sample construction strategy and defect discriminator design technology based on Gaussian noise of the present invention. Detailed Implementation

[0029] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0030] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the system design, feature extraction, multimodal feature fusion, and defect point detection of industrial parts based on 2D-3D multimodal images proposed in this invention.

[0031] like Figure 1 The diagram shows the structure of an industrial parts defect detection system. The system includes an image acquisition device, a data processing module, a feature extraction module, a feature fusion module, and a defect detection module. The image acquisition device includes 2D and 3D cameras, responsible for acquiring 2D image data and 3D point cloud data of the target industrial parts. The data processing module is responsible for image enhancement and preprocessing, including random cropping of the 2D image center and extraction of foreground image content, while filtering and denoising the 3D point cloud data. The feature extraction module uses a convolutional neural network to extract high-level key features from the 2D image data and uses point cloud processing algorithms to extract features from the 3D point cloud data. The feature fusion module fuses the features obtained after processing by the 2D and 3D feature extraction modules. The defect detection module is responsible for defect detection and classification, and accurately locating possible defect points. This system fully integrates 2D and 3D data sources and uses deep learning methods to achieve comprehensive and accurate industrial parts defect detection, improving the accuracy and applicability of defect detection.

[0032] A defect detection method for industrial parts based on 2D-3D multimodal images includes feature extraction design, multimodal feature fusion method, defect discriminator design, and end-to-end detection model training.

[0033] The specific process of feature extraction for multimodal images is as follows:Figure 2 As shown in FIG. 2, the method mainly includes the following steps:

[0034] For the 2D image data sample input to the model, the key features of the 2D image are extracted based on the ResNet class backbone neural network pre-trained by ImageNet. Since the abstract information represented by the extracted deep features is too biased towards the data distribution of ImageNet, in the method, the multi-level feature maps of the intermediate output are selected, and the whole picture is divided into blocks, and the size of each block is patch_size, which is 3. And for the original input after preprocessing, the size of the model is 224x224, and the batch size batch_size is set to 2, so the input size is (2, 3, 224, 224), and the selected multi-level intermediate layers layer1 and layer2 have feature sizes of 28x28 and 14x14 respectively. Linear interpolation is performed on layer2 to unify the size, complete linear mapping, and feature splicing to realize the final 2D feature representation.

[0035] The alignment of 2D and 3D images is realized based on camera parameters, which facilitates 3D feature extraction and feature fusion operations in the corresponding region.

[0036] For the preprocessed 3D point cloud data, FPS farthest point sampling strategy is used to group points, and the size of each group is set to group_size, which is 10. Then the pre-trained PointFormer network on ShapeNet is used to extract features of each group, and the multi-level features are obtained. Linear interpolation is performed on the multi-level features, which are scaled to the same size as the 2D feature map, i.e. 28x28, and then the multi-level features are spliced to obtain the final 3D feature map corresponding to the 2D feature map.

[0037] The multi-modal feature fusion process based on cross-attention mechanism is as shown in FIG. 4, which mainly includes the following steps: Figure 3

[0038] The 2D and 3D input data are obtained by the feature extractor. Although each pair of 2D and 3D images is taken at the same angle, there is still a certain degree of semantic difference between the two modalities. Therefore, these features are sent to the multi-modal feature fusion module, based on the cross-attention mechanism, the attention weight of the 2D picture feature to the 3D point cloud feature is calculated, the weighted 2D picture feature and 3D point cloud feature are spliced, the feature fusion is realized, and the final positive sample feature map real_map is obtained.

[0039] The negative sample construction strategy based on Gaussian noise and the defect discriminator design process are as shown in FIG. 6, which mainly includes the following steps: Figure 4

[0040] ​​In the defect detection application scenario, the real abnormal area and abnormal category are unpredictable, so in order to make the model learn the feature information of the positive sample sufficiently, a large number of positive samples are contained in the training data set. In the model training process, the feature information of the negative sample is derived from the generator introducing Gaussian noise. The Gaussian noise generator can generate random noise in a specific feature space. These noises (the size is consistent with the feature real_feat of the positive sample) are added to the feature representation of the real sample to obtain the negative sample feature fake_feat. These generated negative samples are similar to the real defect samples in feature distribution, so as to ensure the robustness and universality of the detection model in the training process.

[0041] In the training process, the positive sample and the negative sample are input into the defect discriminator, so that the model learns to distinguish the positive and negative sample features. The defect discriminator is a network composed of two layers of multilayer perceptron (MLP), and outputs an evaluation score S of the feature vector. The feature vector with the highest score in each feature map can represent the evaluation score of the entire input sample, that is, S ad =max(S i )i=1,2,...,N, wherein N is the number of feature vectors in the feature map. The more scores higher than the threshold value th (th is set to 0.75), the greater the possibility of the region being a defect point, so a kind of truncated loss function is designed as follows:

[0042] L=max(0,th-D(real_feat))+max(0,th+D(fake_feat))

[0043] Wherein L is the loss corresponding to each feature vector, D is the discriminator, and the overall loss is the sum of the losses corresponding to the feature vectors. After bilinear interpolation, the scores of each region are restored to the heat map of the input image size, realizing the visualization of the defect points.

[0044] The negative sample generator introducing Gaussian noise is only used in the training process, and this link is not needed in the inference and defect detection process. The input sample can be directly detected and evaluated for defects.

[0045] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting defects in industrial parts based on 2D-3D multimodal images, characterized in that, It uses an industrial camera to acquire 2D and 3D images of the parts to be inspected. By comprehensively utilizing the 2D and 3D image data, a deep learning model is used for feature extraction, and feature fusion is used to achieve high-precision and universal defect detection. The industrial parts defect detection method includes the following steps: Acquire 2D image data of the target industrial part from the image acquisition device; acquire depth image and corresponding 3D point cloud data of the target industrial part from the image acquisition device; Image enhancement processing is performed on the acquired 2D image data to improve image quality, and random cropping and foreground extraction are performed to ensure the training effect of the detection model. In addition, the acquired 3D point cloud data is filtered and denoised to reduce data noise; For the preprocessed 2D image data, a feature extractor is designed based on a convolutional neural network to extract features and generate 2D feature vectors; and for the preprocessed 3D point cloud data, a point cloud processing algorithm is used to extract features and generate 3D feature vectors. Feature fusion is performed on 2D and 3D feature vectors to obtain 2D-3D multimodal features; A discriminator model was designed using deep learning technology to detect and classify defects based on 2D-3D multimodal features, and the possible locations of defects were determined based on the comprehensive defect assessment results. The process of extracting features from the preprocessed 2D image data using a feature extractor designed based on a convolutional neural network to generate 2D feature vectors includes: For the preprocessed 2D image, a ResNet-like backbone network pre-trained for image classification is used to extract high-level abstract features from the image. Two feature maps of different intermediate sizes are saved, and linear mapping and feature concatenation are performed to obtain the final 2D feature map. The preprocessed 3D point cloud data is then subjected to feature extraction using a point cloud processing algorithm to generate a 3D feature vector, including: For the preprocessed 3D point cloud data, the FPS farthest sampling technique is used to group the points, and the PointFormer network is used to perform point cloud feature extraction to obtain the 3D feature vector of each group. The feature fusion of 2D and 3D feature vectors includes: The obtained 2D and 3D features are fused using a cross-attention mechanism to obtain the final positive sample feature real_feat, enabling the model to fully learn the positive sample information.

2. The method for detecting defects in industrial parts according to claim 1, characterized in that, The defect detection and classification process includes: While learning from the fused positive samples, the detection network introduces Gaussian noise to generate anomalous samples with different distributions, i.e., negative sample features fake_feat. The model treats the positive and negative samples as 0 and 1 labels, respectively, for training. During the inference detection process, Gaussian noise is no longer introduced. Instead, the detection process is directly performed on the anomalous images. Based on the defect assessment results output by the model, the possible defect locations are determined, and a heatmap is generated to visualize the detection results.

3. An industrial part defect detection system based on 2D-3D multimodal images, characterized in that, The system includes: The image acquisition device module is used to acquire 2D image data of the target industrial part from the image acquisition device; and to acquire depth images and corresponding 3D point cloud data of the target industrial part from the image acquisition device. The data processing module is used to perform image enhancement processing on the acquired 2D image data to improve image quality, and to perform center random cropping and image foreground extraction processing to ensure the training effect of the detection model; and to perform filtering and denoising processing on the acquired 3D point cloud data to reduce data noise. The feature extraction module is used to extract features from preprocessed 2D image data using a feature extractor designed based on a convolutional neural network, generating 2D feature vectors; and to extract features from preprocessed 3D point cloud data using a point cloud processing algorithm, generating 3D feature vectors. The feature fusion module is used to fuse 2D feature vectors and 3D feature vectors to obtain 2D-3D multimodal features; The defect detection module is used to design a discriminator model using deep learning technology, perform defect detection and classification based on 2D-3D multimodal features, and determine the possible defect locations based on the comprehensive defect evaluation results. The process of extracting features from the preprocessed 2D image data using a feature extractor designed based on a convolutional neural network to generate 2D feature vectors includes: For the preprocessed 2D image, a ResNet-like backbone network pre-trained for image classification is used to extract high-level abstract features from the image. Two feature maps of different intermediate sizes are saved, and linear mapping and feature concatenation are performed to obtain the final 2D feature map. The preprocessed 3D point cloud data is then subjected to feature extraction using a point cloud processing algorithm to generate a 3D feature vector, including: For the preprocessed 3D point cloud data, the FPS farthest sampling technique is used to group the points, and the PointFormer network is used to perform point cloud feature extraction to obtain the 3D feature vector of each group. The feature fusion of 2D and 3D feature vectors includes: The obtained 2D and 3D features are fused using a cross-attention mechanism to obtain the final positive sample feature real_feat, enabling the model to fully learn the positive sample information.

4. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the method for detecting defects in industrial parts as described in any one of claims 1 to 2 is performed.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the industrial part defect detection method as described in any one of claims 1 to 2.

Images