Defect detection method and device based on semantic guidance cross-modal feature reconstruction

Abstract

The invention provides a defect detection method and device based on semantic guidance cross-modal feature reconstruction, and belongs to the technical field of industrial intelligent detection.The method comprises the steps that a two-way parallel feature extraction network is constructed, and multi-level features of an input image and point cloud data are obtained; a text perception fusion module based on dynamic graph convolution is added, cross-modal fusion is carried out on text prompts as semantic guidance information and image and point cloud multi-level features, and semantic consistency and expression ability of feature representation are improved; a bidirectional cross-modal feature reconstruction network is built, point cloud reconstruction and image reconstruction branches are included, defect recognition and positioning are achieved through difference analysis of reconstruction features and original features, and finally a defect detection result of multi-dimensional information perception is output; the device comprises a processor and a memory. According to the method, the complementary advantages of three kinds of modal information of the image, the point cloud and the text are fully utilized, and a text-guided cross-modal double-branch feature reconstruction network is constructed to realize accurate detection of target defects.

Description

Technical Field

[0001] This invention belongs to the field of industrial intelligent inspection technology, specifically relating to a defect detection method and device based on semantically guided cross-modal feature reconstruction. Background Technology

[0002] With the rapid development of industrial intelligent inspection technology, traditional defect detection methods mainly rely on single-modal data input, such as images or point cloud data. While these methods can effectively identify defects in certain situations, they are often limited in complex scenarios due to insufficient information between modalities and limitations in feature representation capabilities, resulting in insufficient detection accuracy and robustness. Especially in the processing of multimodal data, how to effectively fuse features from different modalities to improve the accuracy and efficiency of defect detection remains a pressing technical challenge. In recent years, cross-modal learning methods have gradually become an important approach to solving this problem. By effectively fusing data from different modalities, more comprehensive and accurate feature representation can be achieved. However, existing cross-modal learning methods often ignore the potential semantic relationships between different modalities, resulting in fused features that cannot fully reflect the semantic consistency between modalities, thus affecting the final detection performance. Summary of the Invention

[0003] The purpose of this invention is to propose a defect detection method and device based on semantic-guided cross-modal feature reconstruction. It fully utilizes the complementary advantages of image, point cloud, and text information to construct a text-guided cross-modal bi-branch feature reconstruction network to achieve accurate detection of target defects.

[0004] This invention is achieved through the following technical solution:

[0005] A defect detection method based on semantically guided cross-modal feature reconstruction includes the following steps:

[0006] Step S1: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information.

[0007] Step S2: Add a text-aware fusion module based on dynamic graph convolution, jointly construct a joint correlation matrix between nodes by combining text, image, and point cloud features, and use dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, the alignment of features between modalities is optimized, and the semantic consistency and expressive power of feature representation are improved.

[0008] Step S3: Construct a bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor. Input the cross-modal features fused in step S2 into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect area is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

[0009] Furthermore, in step S1, for the input image I and point cloud data P, multi-level features are obtained in parallel through upper and lower path feature extraction networks, specifically including: the upper path according to the formula... Obtain the l-th level visual features F of the input image I l I ,in, This represents the output of the l-th layer of the image feature extractor, where l = 3, 7, 11. The image feature extractor uses the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset; the next path is based on the formula... Obtain the l-th level geometric features F of point cloud data P l P ,in, This represents the output of the l-th layer of the point cloud feature extractor, where l = 3, 7, 11. The point cloud feature extractor uses a Point-MAE network pre-trained on the ShapeNet dataset.

[0010] Furthermore, step S2 specifically includes the following steps:

[0011] Step S21: Design a text prompt template, based on s n =[P1][P2]...[P N [obj.] Design learnable normal prompts, based on s a =[P1][P2]...[P N [obj.][A1][A2]...[A M Design learnable anomaly prompts, where N and M represent the lengths of learnable normal prefixes and learnable anomalous suffixes, respectively, [obj.] represents the text tokens representing the detected object category, and [P i ]、[A k ] represents the learnable normal and abnormal word units, i∈{1,...,n},k∈{1,...,m}. The embedding vectors of these word units are global parameters that are optimized during model training.

[0012] Step S22: The normal prompt s constructed in step S21 n and abnormal prompts a Input text feature extractor, based on F T =Concat(ΦT (s n ),Φ T (s a Obtain all text prompt features F from the input. T , where Φ T This represents a text feature extractor, which employs a text encoder based on the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset;

[0013] Step S23: Process the multi-level features F of the image and point cloud extracted in step S1 using a global average pooling layer. l I F l P Obtain the corresponding global feature representation. Based on the text features F obtained in step S22 T According to the formula Obtain the joint correlation matrix A of complex nonlinear relationships among different modal features. d,l Where δ(·) is the activation function, f conv (·) represents a convolutional layer;

[0014] Step S24: To learn the latent semantics of the text prompts and their correlation with image and point cloud features, text features are used as graph nodes, and the semantic correlation between features is used as graph edge weights, according to the formula... Obtain a text representation with potential visual information perception. Where LeakyReLU(·) is the activation function, W d,l Update the matrix for learnable dynamic weights;

[0015] Step S25: Based on the multi-level image and point cloud features obtained in step S1 and the text representation obtained in step S24, according to the formula... We obtain cross-modal fusion features guided by textual semantics, where softmax(·) is the normalized exponential function and α∈{I,P} is the modality identifier.

[0016] Furthermore, in step S3, the bidirectional cross-modal feature reconstruction network includes a point cloud reconstructor and an image reconstructor, both of which are composed of multilayer perceptrons. The multilayer perceptron sequentially includes: a first linear layer, a first batch normalization layer, a first nonlinear activation layer, a second linear layer, a second batch normalization layer, a second nonlinear activation layer, a third linear layer, a third batch normalization layer, and a third nonlinear activation layer. The input dimension of the first linear layer is the number of channels of the input features, and the output dimension is half the sum of the number of channels of the fused point cloud features and the image features; the input and output dimensions of the second linear layer are consistent; the output dimension of the third linear layer is the number of channels of the target modality features. Through this structure, effective reconstruction of the fused features in their respective modalities is achieved, thereby improving the cross-modal feature restoration capability of defect detection.

[0017] Furthermore, in step S3, the difference analysis between the reconstructed features and the original features is based on the multi-level image fusion features obtained in step S25. Input the point cloud features into the point cloud reconstructor respectively. The obtained multi-level point cloud fusion features Input the image features into the image reconstructor respectively. The difference maps are obtained by calculating the Euclidean distance between the restored point cloud features and the point cloud fusion features, and between the restored image features and the image fusion features. The above difference maps are combined using an aggregation function to output the final detection map Ψ.

[0018] Furthermore, both the images and point cloud data are from the MVTec 3D-AD dataset.

[0019] This invention is also achieved through the following technical solutions:

[0020] A defect detection device based on semantically guided cross-modal feature reconstruction, characterized in that it includes a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:

[0021] Feature extraction module: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information.

[0022] Semantic-guided fusion module: A text-aware fusion module based on dynamic graph convolution is added. It combines text, image, and point cloud features to construct a joint correlation matrix between nodes, and uses dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, it optimizes the alignment of features between modalities and improves the semantic consistency and expressive power of feature representation.

[0023] Cross-modal reconstruction module: A bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor is constructed. The fused cross-modal features are input into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect region is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

[0024] The present invention has the following beneficial effects:

[0025] 1. This invention introduces a text-aware fusion module based on dynamic graph convolution, which can effectively combine text, image and point cloud features and capture the dynamic relationship between the three, effectively improving the consistency of feature representation and semantic expression ability between high modalities;

[0026] 2. This invention differs from the traditional single-modal reconstruction approach by constructing a bidirectional cross-modal feature reconstruction network. The fused image and point cloud features are processed by point cloud reconstructors and image reconstructors respectively, exhibiting high sensitivity to various defect types and good robustness against false detections.

[0027] 3. This invention can comprehensively perceive and process information from different data modalities, providing detection results with multi-dimensional information perception, improving the quality of single-modal feature reconstruction, enhancing robustness, and providing a new methodological approach for solving surface defect detection based on feature reconstruction. Attached Figure Description

[0028] The present invention will now be described in further detail with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0031] like Figure 1 and Figure 2 As shown, a defect detection method based on semantically guided cross-modal feature reconstruction includes the following steps:

[0032] Step S1: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information.

[0033] Specifically, for the input image I and point cloud data P, multi-level features are obtained in parallel through upper and lower path feature extraction networks. The lower-level features contain detailed texture information, while the higher-level features contain contextual semantic information. Specifically, the upper path uses the formula... Obtain the l-th level visual features F of the input image I l I ,in, This represents the output of the l-th layer of the image feature extractor, where l = 3, 7, 11. The image feature extractor uses the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset; the next path is based on the formula... Obtain the l-th level geometric features F of point cloud data P l P ,in, This represents the output of the l-th layer of the point cloud feature extractor, where l = 3, 7, 11. The point cloud feature extractor uses a Point-MAE network pre-trained on the ShapeNet dataset.

[0034] The input images and point cloud data were both derived from the MVTec 3D-AD dataset, which contains 4147 samples across 10 categories, including 2656 defect-free training samples, 294 validation samples, and 1197 test samples. The test samples include both defect-free and defect-free samples. Each sample contains a colored point cloud, and the defect categories cover 41 types, including texture and geometric defects, encompassing multiple detection scenarios such as textures and objects.

[0035] Step S2: Add a text-aware fusion module based on dynamic graph convolution, jointly construct a joint correlation matrix between nodes by combining text, image, and point cloud features, and use dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, the alignment of features between modalities is optimized, and the semantic consistency and expressive power of feature representation are improved.

[0036] Specifically, the steps include the following:

[0037] Step S21: Design a text prompt template, based on s n =[P1][P2]...[P N [obj.] Design learnable normal prompts, based on s a =[P1][P2]...[P N[obj.][A1][A2]...[A M Design learnable anomaly prompts, where N and M represent the lengths of learnable normal prefixes and learnable anomalous suffixes, respectively, [obj.] represents the text tokens representing the detected object category, and [P i ]、[A k ] represents the learnable normal and abnormal word units, i∈{1,...,n},k∈{1,...,m}. The embedding vectors of these word units are global parameters that are optimized during model training.

[0038] Step S22: The normal prompt s constructed in step S21 n and abnormal prompts a Input text feature extractor, based on F T =Concat(Φ T (s n ),Φ T (s a Obtain all text prompt features F from the input. T , where Φ T This represents a text feature extractor, which employs a text encoder based on the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset;

[0039] Step S23: Process the multi-level features F of the image and point cloud extracted in step S1 using a global average pooling layer. l I F l P Obtain the corresponding global feature representation. Based on the text features F obtained in step S22 T According to the formula Obtain the joint correlation matrix A of complex nonlinear relationships among different modal features. d,l Where δ(·) is the activation function, f conv (·) represents a convolutional layer;

[0040] By using a global average pooling layer to extract global information, global semantic relationships between different modalities can be effectively captured. This operation helps to better combine image and point cloud features with text features, thereby constructing a joint correlation matrix of complex nonlinear relationships between features of different modalities, and further improving the effect of cross-modal feature alignment.

[0041] Step S24: To learn the latent semantics of the text prompts and their correlation with image and point cloud features, text features are used as graph nodes, and the semantic correlation between features is used as graph edge weights, according to the formula... Obtain a text representation with potential visual information perception. Where LeakyReLU(·) is the activation function, W d,l Update the matrix for learnable dynamic weights;

[0042] Step S25: Based on the multi-level image and point cloud features obtained in step S1 and the text representation obtained in step S24, according to the formula... We obtain cross-modal fusion features guided by textual semantics, where softmax(·) is the normalized exponential function and α∈{I,P} is the modality identifier.

[0043] Step S3: Construct a bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor. Input the cross-modal features fused in step S2 into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect area is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

[0044] Specifically, the bidirectional cross-modal feature reconstruction network consists of two parts: a point cloud reconstructor and an image reconstructor. Both the point cloud reconstructor and the image reconstructor are constructed using multilayer perceptrons. The multilayer perceptron sequentially includes: a first linear layer, a first batch normalization layer, a first nonlinear activation layer, a second linear layer, a second batch normalization layer, a second nonlinear activation layer, a third linear layer, a third batch normalization layer, and a third nonlinear activation layer. The input dimension of the first linear layer is the number of channels of the input features, and the output dimension is half the sum of the number of channels of the fused point cloud features and image features. The input and output dimensions of the second linear layer are consistent. The output dimension of the third linear layer is the number of channels of the target modal features.

[0045] Specifically, the point cloud reconstructor uses the multi-level image fusion features obtained in step S25. As input, reconstruct point cloud features The image reconstructor uses the multi-level point cloud fusion features obtained in step S25 As input, restore image features This bidirectional reconstruction paradigm forces the model to simultaneously learn the mapping relationship between image features and point cloud features, achieving precise alignment of features of different modalities in the semantic space. Simultaneously, the difference maps are obtained by calculating the Euclidean distances between the reconstructed point cloud features and the fused point cloud features, and between the reconstructed image features and the fused image features. According to the formula By aggregating the above discrepancies, the input size is restored through bilinear interpolation, and the final output is the detection map Ψ.

[0046] During the training phase, only normal samples are used for training. The cross-modal reconstruction loss function of the l-th layer is expressed as: The total loss function obtained by summing the reconstruction losses of the three layers is expressed as L = L1 + L2 + L3, where h l w l These represent the height and width of the image feature at level l, respectively. Representing reconstructed image features Reconstructing point cloud features The feature block located at coordinates (m, n) Representing point cloud fusion features Image fusion features The feature block located at coordinates (m,n), where m∈{1,2,…,h} l}, n∈{1,2,…,w l}

[0047] The reconstructed features of normal samples have a high cosine similarity to the input features, meaning the model can reconstruct the features of normal samples. In contrast, the reconstructed features of abnormal samples have a low cosine similarity to the input features, meaning the reconstruction error of abnormal samples is large. Therefore, the model can detect defects based on the error in feature reconstruction.

[0048] Correspondingly, the defect detection device based on semantically guided cross-modal feature reconstruction includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:

[0049] Feature extraction module: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information.

[0050] Semantic-guided fusion module: A text-aware fusion module based on dynamic graph convolution is added. It combines text, image, and point cloud features to construct a joint correlation matrix between nodes, and uses dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, it optimizes the alignment of features between modalities and improves the semantic consistency and expressive power of feature representation.

[0051] Cross-modal reconstruction module: A bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor is constructed. The fused cross-modal features are input into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect region is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

[0052] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the specification of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A defect detection method based on semantically guided cross-modal feature reconstruction, characterized in that, Includes the following steps: Step S1: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information. Step S2: Add a text-aware fusion module based on dynamic graph convolution, jointly construct a joint correlation matrix between nodes by combining text, image, and point cloud features, and use dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, the alignment of features between modalities is optimized, and the semantic consistency and expressive power of feature representation are improved. Step S3: Construct a bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor. Input the cross-modal features fused in step S2 into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect area is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

2. The defect detection method based on semantically guided cross-modal feature reconstruction according to claim 1, characterized in that: In step S1, for the input image I and point cloud data P, multi-level features are obtained in parallel through upper and lower path feature extraction networks. Specifically, this includes: the upper path extracts features according to the formula... Obtain the l-th level visual features F of the input image I l I ,in, This represents the output of the l-th layer of the image feature extractor, where l = 3, 7, 11. The image feature extractor uses the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset; the next path is based on the formula... Obtain the l-th level geometric features F of point cloud data P l P ,in, This represents the output of the l-th layer of the point cloud feature extractor, where l = 3, 7, 11. The point cloud feature extractor uses a Point-MAE network pre-trained on the ShapeNet dataset.

3. The defect detection method based on semantically guided cross-modal feature reconstruction according to claim 1, characterized in that: Step S2 specifically includes the following steps: Step S21: Design a text prompt template, based on s n =[P1][P2]...[P N [obj.] Design learnable normal prompts, based on s a =[P1][P2]...[P N [obj.][A1][A2]...[A M Design learnable anomaly prompts, where N and M represent the lengths of learnable normal prefixes and learnable anomalous suffixes, respectively, [obj.] represents the text tokens representing the detected object category, and [P i ]、[A k ] represents the learnable normal and abnormal word units, i∈{1,...,n},k∈{1,...,m}. The embedding vectors of these word units are global parameters that are optimized during model training. Step S22: The normal prompt s constructed in step S21 n and abnormal prompts a Input text feature extractor, based on F T =Concat(Φ T (s n ),Φ T (s a Obtain all input text prompt features F T , where Φ T This represents a text feature extractor, which employs a text encoder based on the ViT-B / 16CLIP model pre-trained on the LAION-400M dataset; Step S23: Process the multi-level features F of the image and point cloud extracted in step S1 using a global average pooling layer. l I F l P Obtain the corresponding global feature representation. Based on the text features F obtained in step S22 T According to the formula Obtain the joint correlation matrix A of complex nonlinear relationships among different modal features. d,l Where δ(·) is the activation function, f conv (·) represents a convolutional layer; Step S24: To learn the latent semantics of the text prompts and their correlation with image and point cloud features, text features are used as graph nodes, and the semantic correlation between features is used as graph edge weights, according to the formula... Obtain a text representation with potential visual information perception. Where LeakyReLU(·) is the activation function, W d,l Update the matrix for learnable dynamic weights; Step S25: Based on the multi-level image and point cloud features obtained in step S1 and the text representation obtained in step S24, according to the formula... We obtain cross-modal fusion features guided by textual semantics, where softmax(·) is the normalized exponential function and α∈{I,P} is the modality identifier.

4. The defect detection method based on semantically guided cross-modal feature reconstruction according to claim 1, characterized in that: In step S3, the bidirectional cross-modal feature reconstruction network includes a point cloud reconstructor and an image reconstructor, both of which are composed of multilayer perceptrons. The multilayer perceptron sequentially includes: a first linear layer, a first batch normalization layer, a first nonlinear activation layer, a second linear layer, a second batch normalization layer, a second nonlinear activation layer, a third linear layer, a third batch normalization layer, and a third nonlinear activation layer. The input dimension of the first linear layer is the number of channels of the input features, and the output dimension is half the sum of the number of channels of the fused point cloud features and the image features. The input and output dimensions of the second linear layer are consistent. The output dimension of the third linear layer is the number of channels of the target modality features. Through this structure, effective reconstruction of the fused features in their respective modalities is achieved, thereby improving the cross-modal feature restoration capability of defect detection.

5. The defect detection method based on semantically guided cross-modal feature reconstruction according to claim 1, characterized in that: In step S3, the difference analysis between the reconstructed features and the original features is based on the multi-level image fusion features obtained in step S25. Input the point cloud features into the point cloud reconstructor respectively. The obtained multi-level point cloud fusion features Input the image features into the image reconstructor respectively. The difference maps are obtained by calculating the Euclidean distance between the restored point cloud features and the point cloud fusion features, and between the restored image features and the image fusion features. The above difference maps are combined using an aggregation function to output the final detection map Ψ.

6. A defect detection method based on semantically guided cross-modal feature reconstruction according to any one of claims 1 to 5, characterized in that: The images and point cloud data are both from the MVTec 3D-AD dataset.

7. A defect detection device based on semantically guided cross-modal feature reconstruction, characterized in that: It includes a processor and memory. The memory stores program instructions, and the processor calls the program instructions in memory to implement the following modules: Feature extraction module: Construct a dual-path parallel feature extraction network to obtain multi-level features of the input image and point cloud data. The upper path network extracts the visual features of the image, and the lower path network extracts the geometric features of the point cloud. The extracted multi-level features have rich detailed information and contextual semantic information. Semantic-guided fusion module: A text-aware fusion module based on dynamic graph convolution is added. It combines text, image, and point cloud features to construct a joint correlation matrix between nodes, and uses dynamic graph convolution to capture the dynamic relationship between the three. By fusing semantic guidance information across modalities, it optimizes the alignment of features between modalities and improves the semantic consistency and expressive power of feature representation. Cross-modal reconstruction module: A bidirectional cross-modal feature reconstruction network consisting of a point cloud reconstructor and an image reconstructor is constructed. The fused cross-modal features are input into the point cloud reconstruction branch and the image reconstruction branch respectively for feature reconstruction. By analyzing the difference between the reconstructed features and the original features, the defect region is identified and located. Finally, the defect detection result with multi-dimensional information perception is output.

Images