Multimodal industrial anomaly detection method and device based on bidirectional feature mapping

Abstract

The invention discloses a multimodal industrial anomaly detection method and device based on bidirectional feature mapping, and relates to the technical field of multimodal industrial anomaly detection, and the method comprises the steps: extracting a to-be-processed image modal feature corresponding to the image modal of a product and a to-be-processed point cloud modal feature corresponding to the point cloud modal of the product; performing cross-modal feature prediction on the to-be-processed image modal features and the to-be-processed point cloud modal features based on a preset cross-modal feature prediction task to obtain a first feature prediction result and a second feature prediction result; reconstructing the modal features of the to-be-processed image and the modal features of the to-be-processed point cloud based on a feature reconstruction task in a preset modal to obtain an image modal feature reconstruction result and a point cloud modal feature reconstruction result; and determining an anomaly detection result based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result. In this way, the accuracy of product anomaly detection can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal industrial anomaly detection, and in particular to a multimodal industrial anomaly detection method and device based on bidirectional feature mapping. Background Art

[0002] Industrial anomaly detection is an important part of industrial automation production. It plays an important role in improving the efficiency of industrial product detection and reducing the problem of false detection caused by worker fatigue. Due to the particularity of the industrial production environment, industrial products present various forms of anomalies, and products of specific anomaly types are not easy to obtain. Therefore, in recent years, research on industrial anomaly detection tasks has largely adopted unsupervised methods to detect anomalies of industrial products.

[0003] At present, industrial anomaly detection is mainly studied based on RGB image data. In the prior art, RGB image anomaly detection methods are mainly based on pre-trained features and reconstruction-based methods. However, it is often difficult to accurately identify three-dimensional features such as volume defects and structural anomalies of products by relying solely on RGB image data. Therefore, introducing the point cloud modal information of the target product into the anomaly detection task shows better anomaly detection performance than using only the image modality of the target product.

[0004] In multimodal industrial anomaly detection technology, the solution is to adopt a feature-based detection paradigm. This solution usually includes two steps: first, multimodal features are extracted using a pre-trained model, and then anomaly detection is performed based on the extracted features. After obtaining the multimodal features, the feature distribution model of normal samples is mainly established by means of a feature repository. This method has technical problems such as low space utilization efficiency of the feature repository, high inference calculation overhead, and the need for a large number of normal samples.

[0005] Although the cross-modal feature prediction method proposed at present has improved the above problems to a certain extent, the method still faces technical difficulties in dealing with single-modal anomalies. For example, slight contamination on the surface of the product is obvious in 3D data, but difficult to detect in RGB images, or the color anomaly of the product is easy to identify in RGB images, but cannot be reflected in point cloud data. In addition, due to the inherent feature mapping asymmetry between different modalities, the features of one modality may correspond to multiple different features in another modality. This one-to-many mapping relationship exists not only in abnormal samples, but also in normal samples. For example, due to changes in lighting conditions and shooting angles, the RGB image features of the same product may change significantly, while its 3D features remain unchanged. The uncertainty of this feature mapping makes it difficult for the anomaly detection model to accurately predict the cross-modal feature correspondence, which is easy to cause false alarms and reduce the detection accuracy.

[0006] It can be seen from the above that how to improve the accuracy of abnormality detection of products is a problem that needs to be solved urgently. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a multimodal industrial anomaly detection method and device based on bidirectional feature mapping, which can determine the anomaly detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result, thereby improving the accuracy and efficiency of anomaly detection of the product. The specific scheme is as follows:

[0008] In a first aspect, the present application provides a multimodal industrial anomaly detection method based on bidirectional feature mapping, comprising:

[0009] Collecting image modalities and point cloud modalities corresponding to the product to be inspected, and using a first preset feature extractor to extract two-dimensional features corresponding to the image modality to obtain image modal features to be processed, and then using a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed;

[0010] Based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the image to be processed and the modal features of the point cloud to be processed to obtain corresponding first feature prediction results and second feature prediction results;

[0011] Reconstructing the image modality features to be processed and the point cloud modality features to be processed respectively based on the preset intra-modality feature reconstruction task in the bidirectional feature mapping framework, and obtaining corresponding image modality feature reconstruction results and point cloud modality feature reconstruction results respectively;

[0012] An abnormality detection result corresponding to the product to be detected is determined based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result.

[0013] Optionally, the collecting of image modalities and point cloud modalities corresponding to the product to be inspected, and using a first preset feature extractor to extract two-dimensional features corresponding to the image modalities to obtain image modal features to be processed, includes:

[0014] Collecting image modalities and point cloud modalities corresponding to the product to be inspected; the image modalities and the point cloud modalities are mapped one by one on pixels;

[0015] Using a first preset feature extractor to divide the image modality into a number of image blocks of the same size according to a preset image division rule, and performing feature extraction on each of the image blocks to obtain corresponding first feature information;

[0016] Each of the first feature information is processed using a preset bilinear interpolation technology to obtain a modal feature of the image to be processed; the modal feature of the image to be processed is a two-dimensional feature.

[0017] Optionally, the extracting three-dimensional features corresponding to the point cloud modality using a second preset feature extractor to obtain corresponding point cloud modality features to be processed includes:

[0018] Processing the point cloud modality using the second preset feature extractor and based on a preset sampling method to obtain a plurality of point groups;

[0019] Feature extraction is performed on each of the point groups to obtain corresponding second feature information, and each of the second feature information is processed using a preset point feature interpolation method to obtain a point cloud modal feature to be processed; the point cloud modal feature to be processed is a three-dimensional feature.

[0020] Optionally, performing cross-modal feature prediction on the modal features of the image to be processed based on a preset cross-modal feature prediction task in the bidirectional feature mapping framework to obtain a corresponding first feature prediction result includes:

[0021] Using the first preset feature prediction mapping function in the preset cross-modal feature prediction task, each of the image modal features to be processed is converted to the corresponding point cloud modal features to be processed for cross-modal feature prediction to obtain a corresponding first feature prediction result;

[0022] Correspondingly, based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the point cloud to be processed to obtain the corresponding second feature prediction result, including:

[0023] By utilizing the second preset feature prediction mapping function in the preset cross-modal feature prediction task, each of the to-be-processed point cloud modal features is converted to the corresponding to-be-processed image modal features for cross-modal feature prediction to obtain a corresponding second feature prediction result.

[0024] Optionally, the using of the first preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each of the image modal features to be processed to the corresponding point cloud modal features to be processed to obtain a corresponding first feature prediction result includes:

[0025] Using the first encoder in the preset cross-modal feature prediction task to encode each of the to-be-processed image modal features to the corresponding to the to-be-processed point cloud modal features to obtain a first encoding result, and then using the first decoder in the preset cross-modal feature prediction task to decode the first encoding result to obtain a corresponding first feature prediction result;

[0026] Correspondingly, the second preset feature prediction mapping function in the preset cross-modal feature prediction task is used to perform cross-modal feature prediction on each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features to obtain a corresponding second feature prediction result, including:

[0027] Using the second encoder in the preset cross-modal feature prediction task to encode each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features to obtain a second encoding result, and then using the second decoder in the preset cross-modal feature prediction task to decode the second encoding result to obtain a corresponding second feature prediction result;

[0028] Correspondingly, the preset intra-modality feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modality features of the image to be processed to obtain the corresponding image modality feature reconstruction result, including:

[0029] Using the first encoder and based on a first preset feature reconstruction mapping function, each of the image modality features to be processed is encoded respectively to obtain a third encoding result, and then using the first decoder to decode the third encoding result to obtain a corresponding image modality feature reconstruction result;

[0030] Correspondingly, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modal features of the point cloud to be processed, and the corresponding point cloud modal feature reconstruction result is obtained, including:

[0031] The second encoder is used to encode each of the point cloud modal features to be processed based on a second preset feature reconstruction mapping function to obtain a fourth encoding result, and then the fourth encoding result is decoded by the second decoder to obtain a corresponding image modal feature reconstruction result.

[0032] Optionally, before performing cross-modal feature prediction on the to-be-processed image modal features and the to-be-processed point cloud modal features using the preset cross-modal feature prediction task in the bidirectional feature mapping framework, the method further includes:

[0033] The image modalities and point cloud modalities corresponding to a number of standard products are processed using the bidirectional feature mapping framework to obtain corresponding third feature prediction results, fourth feature prediction results, first feature reconstruction results, and second feature reconstruction results;

[0034] Determine a first loss result based on a similarity distance corresponding to the third feature prediction result and a similarity distance corresponding to the fourth feature prediction result;

[0035] Determine a second loss result based on a similarity distance corresponding to the first feature reconstruction result and a similarity distance corresponding to the second feature reconstruction result;

[0036] A weight coefficient is set based on user needs, and a target loss result is determined using the weight coefficient based on the first loss result and the second loss result, so as to use the target loss result to adjust the parameters corresponding to the preset cross-modal feature prediction task and the preset intra-modal feature reconstruction task.

[0037] Optionally, determining the abnormality detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result includes:

[0038] Determine a corresponding first anomaly score map, a second anomaly score map, a third anomaly score map, and a fourth anomaly score map based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, respectively;

[0039] Using a preset calculation algorithm to calculate the first abnormal score map, the second abnormal score map, the third abnormal score map, and the fourth abnormal score map according to pixel points to obtain an abnormal score map to be processed;

[0040] The to-be-processed anomaly score map is processed using a preset Gaussian smoothing technique to obtain a target anomaly score map, and the anomaly score with the largest value in the target anomaly score map is set as the anomaly detection result corresponding to the to-be-detected product.

[0041] In a second aspect, the present application provides a multimodal industrial anomaly detection device based on bidirectional feature mapping, comprising:

[0042] A feature extraction module, used to collect image modalities and point cloud modalities corresponding to the product to be inspected, and use a first preset feature extractor to extract two-dimensional features corresponding to the image modality to obtain image modal features to be processed, and then use a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed;

[0043] A feature prediction module, used to perform cross-modal feature prediction on the image modality feature to be processed and the point cloud modality feature to be processed based on a preset cross-modal feature prediction task in the bidirectional feature mapping framework, and obtain a corresponding first feature prediction result and a second feature prediction result;

[0044] A feature reconstruction module, used to reconstruct the image modal features to be processed and the point cloud modal features to be processed respectively based on the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework, and obtain corresponding image modal feature reconstruction results and point cloud modal feature reconstruction results respectively;

[0045] A detection result determination module is used to determine an abnormality detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result.

[0046] Optionally, the feature prediction module includes:

[0047] A first encoding result decoding unit, used for encoding each of the to-be-processed image modal features to the corresponding to the to-be-processed point cloud modal features using the first encoder in the preset cross-modal feature prediction task to obtain a first encoding result, and then decoding the first encoding result using the first decoder in the preset cross-modal feature prediction task to obtain a corresponding first feature prediction result;

[0048] Accordingly, the feature prediction module includes:

[0049] A second encoding result decoding unit is used to encode each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features using the second encoder in the preset cross-modal feature prediction task to obtain a second encoding result, and then decode the second encoding result using the second decoder in the preset cross-modal feature prediction task to obtain a corresponding second feature prediction result;

[0050] Accordingly, the feature reconstruction module includes:

[0051] a third encoding result decoding unit, configured to respectively encode each of the to-be-processed image modality features using the first encoder and based on a first preset feature reconstruction mapping function to obtain a third encoding result, and then decode the third encoding result using the first decoder to obtain a corresponding image modality feature reconstruction result;

[0052] Accordingly, the feature reconstruction module includes:

[0053] The fourth encoding result decoding unit is used to use the second encoder and based on the second preset feature reconstruction mapping function to encode each of the to-be-processed point cloud modal features respectively to obtain a fourth encoding result, and then use the second decoder to decode the fourth encoding result to obtain a corresponding image modal feature reconstruction result.

[0054] Optionally, the detection result determination module includes:

[0055] an abnormal score map determining unit, configured to determine a corresponding first abnormal score map, a second abnormal score map, a third abnormal score map, and a fourth abnormal score map based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, respectively;

[0056] an abnormal score map operation unit, configured to use a preset operation algorithm to operate the first abnormal score map, the second abnormal score map, the third abnormal score map, and the fourth abnormal score map according to pixel points to obtain an abnormal score map to be processed;

[0057] The abnormality detection result determination unit is used to process the abnormality score map to be processed using a preset Gaussian smoothing processing technology to obtain a target abnormality score map, and set the abnormality score with the largest value in the target abnormality score map as the abnormality detection result corresponding to the product to be detected.

[0058] As can be seen from the above, before performing multimodal industrial anomaly detection based on bidirectional feature mapping, the present application needs to collect image modalities and point cloud modalities corresponding to the product to be detected, and use a first preset feature extractor to extract two-dimensional features corresponding to the image modality to obtain image modal features to be processed, and then use a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed; use the preset cross-modal feature prediction task in the bidirectional feature mapping framework to perform cross-modal feature prediction on the image modal features to be processed and the point cloud modal features to be processed to obtain corresponding first feature prediction results and second feature prediction results; use the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework to reconstruct the image modal features to be processed and the point cloud modal features to be processed, respectively, to obtain corresponding image modal feature reconstruction results and point cloud modal feature reconstruction results, respectively; determine the anomaly detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result.

[0059] It can be seen that before the multimodal industrial anomaly detection based on bidirectional feature mapping, the present application first needs to collect the image modality and point cloud modality corresponding to the product to be detected, and use the first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use the second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed. Subsequently, the preset cross-modal feature prediction task in the bidirectional feature mapping framework is used to perform cross-modal feature prediction on the image modality features to be processed and the point cloud modality features to be processed, and the corresponding first feature prediction results and second feature prediction results are obtained. Then, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the image modality features to be processed and the point cloud modality features to be processed, respectively, and the corresponding image modality feature reconstruction results and point cloud modality feature reconstruction results are obtained respectively. Finally, the abnormal detection result corresponding to the product to be detected is determined based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result and the point cloud modality feature reconstruction result. In this way, the accuracy and efficiency of abnormality detection of products are improved, thereby further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0061] Figure 1 A flow chart of a multimodal industrial anomaly detection method based on bidirectional feature mapping disclosed in this application;

[0062] Figure 2 A schematic diagram of a process flow between each decoder and each encoder disclosed in the present application;

[0063] Figure 3 A specific multimodal industrial anomaly detection flow chart based on bidirectional feature mapping disclosed in this application;

[0064] Figure 4 This is a schematic diagram of the structure of a multimodal industrial anomaly detection device based on bidirectional feature mapping disclosed in this application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Industrial anomaly detection is an important part of industrial automated production, and plays an important role in improving the efficiency of industrial product detection and reducing the problem of false detection caused by worker fatigue. Due to the particularity of the industrial production environment, industrial products present various forms of anomalies, and products of specific abnormal types are not easy to obtain. Therefore, in recent years, research on industrial anomaly detection tasks has adopted a large number of unsupervised methods to perform anomaly detection of industrial products. To this end, the present application provides a multimodal industrial anomaly detection method based on bidirectional feature mapping, which can determine the anomaly detection results corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result, and the point cloud modal feature reconstruction result, thereby improving the accuracy and efficiency of anomaly detection of products.

[0067] See also Figure 1 As shown, the embodiment of the present invention discloses a multimodal industrial anomaly detection method based on bidirectional feature mapping, including:

[0068] Step S11, collect the image modality and point cloud modality corresponding to the product to be inspected, and use the first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use the second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed.

[0069] In this embodiment, in order to ensure that when multimodal features are extracted for the product to be inspected, not only the point cloud modal features can be interpolated according to their corresponding image positions, but also the image modal features and point cloud modal features in the multimodal sample can be aligned, the RGB image modality and point cloud modality in the multimodal sample need to be mapped one-to-one in pixels.

[0070] It is worth mentioning that after obtaining the image modality and point cloud modality corresponding to the product to be inspected, the embodiment of the present application needs to use a first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use a second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed. Specifically, the image modality and point cloud modality corresponding to the product to be inspected are collected, and the two-dimensional features corresponding to the image modality are extracted using a first preset feature extractor to obtain the image modality features to be processed, and then the three-dimensional features corresponding to the point cloud modality are extracted using a second preset feature extractor to obtain the corresponding point cloud modality features to be processed. This may include: collecting the image modality and point cloud modality corresponding to the product to be inspected; mapping the image modality and the point cloud modality one by one on pixels; using the first preset feature extractor to divide the image modality into several image blocks of the same size according to a preset image division rule, and performing feature extraction on each image block to obtain the corresponding first feature information; using a preset bilinear interpolation technique to process each first feature information to obtain the image modality features to be processed; the image modality features to be processed are two-dimensional features.

[0071] Furthermore, using a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed can include: using the second preset feature extractor and processing the point cloud modality based on a preset sampling method to obtain a number of point groups; performing feature extraction on each point group to obtain corresponding second feature information, and using a preset point feature interpolation method to process each second feature information to obtain the point cloud modal features to be processed; the point cloud modal features to be processed are three-dimensional features.

[0072] In a specific embodiment, the first preset feature extractor may be a pre-trained model DINO (Self-Distillation with No Labels) based on Vision Transformer (i.e., a neural network model based on the Transformer architecture), and the first preset feature extractor is an extractor for performing two-dimensional feature extraction on the image modality corresponding to the product to be inspected. Among them, the process of extracting features from the image modality using the first preset feature extractor is as follows: the image modality is divided into image blocks of a fixed size, and feature extraction is performed on each of the obtained image blocks to obtain the corresponding first feature information, and the first feature information is up-sampled by bilinear interpolation to restore the first feature information to the original image resolution, and finally the image modality feature to be processed is obtained. In a specific embodiment, the image modality feature to be processed may be an RGB (Red, Green, and Blue) modality feature vector.

[0073] Furthermore, the second preset feature extractor may be a pre-trained model PointMAE (i.e., a self-supervised learning model based on point cloud data) based on Point Transformer (i.e., a point cloud processing network based on Transformer), and the second preset feature extractor is an extractor for extracting three-dimensional features of the point cloud modality corresponding to the product to be inspected. The process of extracting features from the point cloud modality using the second preset feature extractor is as follows: first, the point cloud is downsampled into a fixed number of point groups using the farthest point sampling method, and then, feature extraction is performed on the obtained point groups, and the features are restored to the original point cloud resolution using the point feature interpolation method to obtain the features of the point cloud modality to be processed. It is worth mentioning that the point feature interpolation method is used to interpolate the extracted feature values ​​from the point group to each point of the original point cloud. In this way, even if some points in the original point cloud do not appear in the point group, the corresponding feature values ​​can be obtained by the interpolation method.

[0074] It is worth mentioning that the embodiment of the present application utilizes the first preset feature extractor and the second preset feature extractor to perform feature extraction on the image modality and the point cloud modality corresponding to the product to be inspected, respectively, and realizes feature alignment. The purpose is to make full use of the color texture information of the RGB image and the three-dimensional geometric information of the point cloud, thereby improving the accuracy and robustness of anomaly detection.

[0075] In a specific implementation, for the RGB image modality, the embodiment of the present application first evenly divides the input image into image blocks of size p, and then uses the DINO feature extractor to extract features from the RGB image modality to obtain a feature map of size (H / p)×(W / p)×C_2d, and finally uses a bilinear interpolation method to upsample the feature map to the original image resolution H×W×C_2d.

[0076] It is understandable that DINO is a pre-trained model based on Vision Transformer. Among them, Vision Transformer consists of an embedding layer and a Transformer (i.e., a deep learning model based on an attention mechanism) encoder. Furthermore, the embedding layer is used to map the segmented image block sequence and its position information into the input feature vector of the Transformer encoder, and the Transformer encoder is formed by cascading several Transformer blocks with the same structure. It is worth mentioning that since the self-attention mechanism can effectively capture the structural and semantic dependencies between image blocks, each Transformer block consists of a multi-head attention layer and a feedforward layer. The calculation method of the multi-head attention layer is as follows:

[0077] ;

[0078] Where Q is the query vector; K is the key vector, which matches the query vector Q; V is the value vector; d k Represents the dimension of the key vector K.

[0079] Furthermore, the feed-forward layer contains two linear layers and a nonlinear activation function for nonlinear feature transformation, and each Transformer block contains residual connections and layer normalization mechanisms.

[0080] In a specific implementation, for the point cloud modality, the embodiment of the present application first uses the farthest point sampling method to group the point cloud modality to obtain M groups of sampling points, and then uses the PointMAE feature extractor to extract features from the M groups of sampling points respectively to obtain M×C_3d dimensional features, and finally uses the feature interpolation method to restore the obtained three-dimensional features to the original point cloud resolution N×C_3d.

[0081] It can be understood that PointMAE is a pre-trained model based on Point Transformer. Among them, Point Transformer consists of an embedding layer and a Transformer encoder. Furthermore, the embedding layer is used to map the point cloud block sequence and its center point position information into the input feature vector of the Transformer encoder. In addition, the structure of the Transformer encoder is similar to the corresponding structure of the Vision Transformer mentioned above, and will not be described one by one here.

[0082] It is worth mentioning that, in a specific implementation, since the point cloud data is data collected by an RGB-D (i.e., an image representation method that combines color information and depth information) camera, each 3D (3 Dimensions, i.e., three-dimensional) point corresponds one-to-one to a pixel in the RGB image, so the total number of points N is equal to the number of pixels H×W in the RGB image. In addition, to improve computational efficiency, the embodiment of the present application may set the features output by the shallow layer of the Transformer as the modal features of the point cloud to be processed. In a specific implementation, the features output by the shallow layer of the Transformer may be the features output by the first eight layers, which are not specifically limited here.

[0083] Step S12: Based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the image to be processed and the modal features of the point cloud to be processed to obtain corresponding first feature prediction results and second feature prediction results.

[0084] In this embodiment, the bidirectional feature mapping framework includes two feature mapping tasks, cross-modal feature prediction and intra-modal feature reconstruction, which are used to detect the mapped features after stage calculation of cross-modal prediction and intra-modal feature reconstruction. Among them, the cross-modal feature prediction task is used for feature prediction from the modal features of the image to be processed to the modal features of the point cloud to be processed, and from the modal features of the point cloud to be processed to the modal features of the image to be processed.

[0085] It is worth mentioning that based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the image to be processed to obtain the corresponding first feature prediction result, which may include: using the first preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each modal feature of the image to be processed to the corresponding modal feature of the point cloud to be processed to obtain the corresponding first feature prediction result.

[0086] Furthermore, based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the point cloud to be processed to obtain the corresponding second feature prediction results, which may include: using the second preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each modal feature of the point cloud to be processed to the corresponding modal feature of the image to be processed to obtain the corresponding second feature prediction result.

[0087] Furthermore, the cross-modal prediction task establishes a mapping relationship between different modal features by mapping from the source modality to the target modality. In a specific implementation, taking the cross-modal prediction task of mapping the modal features of the image to be processed to the modal features of the point cloud to be processed as an example, the goal is to find the target feature mapping function , so that the similarity distance between the mapped feature and the target feature is minimized. The expression of the target feature mapping function is as follows:

[0088] ;

[0089] in, is the feature vector corresponding to the modal features of the image to be processed, is the feature vector corresponding to the modal feature of the point cloud to be processed, and D represents the similarity distance metric. In a specific implementation, the embodiment of the present application uses cosine distance as the similarity distance metric. Similarly, the cross-modal prediction mapping function from the modal feature of the point cloud to be processed to the modal feature of the image to be processed is It can also be defined in the above manner, which will not be described in detail here.

[0090] Step S13, based on the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework, the image modal features to be processed and the point cloud modal features to be processed are reconstructed respectively to obtain corresponding image modal feature reconstruction results and point cloud modal feature reconstruction results respectively.

[0091] In this embodiment, the intra-modality feature reconstruction task is used to reconstruct the modal features of the image to be processed and the modal features of the point cloud to be processed. It can be understood that the intra-modality feature reconstruction task learns the inherent feature pattern of a single modality by performing feature mapping within the same modality.

[0092] In this embodiment, taking the reconstruction task corresponding to the modal features of the point cloud to be processed as an example, the goal is to find the target feature reconstruction mapping function , so that the similarity distance between the reconstructed features and the original features is minimized. The expression of the target feature reconstruction mapping function is as follows:

[0093] ;

[0094] in, represents the feature vector corresponding to the modal feature of the point cloud to be processed, and D represents the similarity distance metric that is the same as the cross-modal prediction task. Similarly, the reconstruction mapping function corresponding to the modal feature of the image to be processed is The above-mentioned method can also be used for definition, which will not be described in detail here.

[0095] It is understandable that by jointly training the above feature mapping tasks, the bidirectional feature mapping framework can simultaneously optimize its own parameters by using the feature distribution laws of standard samples between modalities and within modalities, so as to use the optimized bidirectional feature mapping framework to perform anomaly detection on the product to be tested. When the similarity between the mapping result and the observed feature is low, it indicates that the current sample may be abnormal. For example, when the cross-modal prediction result is and When the similarity between them is low, it indicates that there is inconsistency between the above samples in the image modality and the point cloud modality.

[0096] It is worth mentioning that the embodiment of the present application constructs an encoder of two shared components and a decoder of two shared components, wherein the encoder is responsible for mapping the features of the source modality to the intermediate features, and the decoder is responsible for mapping the intermediate features back to the target modality space, such as Figure 2As shown. Among them, the feature prediction task and reconstruction task of each modality share the same encoder, and each decoder is used to perform feature prediction tasks and feature reconstruction tasks on the target modality at the same time. Specifically, using the first preset feature prediction mapping function in the preset cross-modal feature prediction task, each image modal feature to be processed is converted to the corresponding point cloud modal feature to be processed for cross-modal feature prediction to obtain the corresponding first feature prediction result, which may include: using the first encoder in the preset cross-modal feature prediction task to encode each image modal feature to be processed to the corresponding point cloud modal feature to be processed to obtain a first encoding result, and then using the first decoder in the preset cross-modal feature prediction task to decode the first encoding result to obtain the corresponding first feature prediction result.

[0097] Correspondingly, using the second preset feature prediction mapping function in the preset cross-modal feature prediction task, cross-modal feature prediction is performed on each point cloud modal feature to be processed to the corresponding image modal feature to be processed to obtain a corresponding second feature prediction result, which may include: using the second encoder in the preset cross-modal feature prediction task to encode each point cloud modal feature to be processed to the corresponding image modal feature to be processed to obtain a second encoding result, and then using the second decoder in the preset cross-modal feature prediction task to decode the second encoding result to obtain a corresponding second feature prediction result.

[0098] Correspondingly, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modal features of the image to be processed to obtain the corresponding image modal feature reconstruction results, which may include: using the first encoder and based on the first preset feature reconstruction mapping function to encode each image modal feature to be processed respectively to obtain a third encoding result, and then using the first decoder to decode the third encoding result to obtain the corresponding image modal feature reconstruction result.

[0099] Correspondingly, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modal features of the point cloud to be processed to obtain the corresponding point cloud modal feature reconstruction results, which may include: using a second encoder and based on a second preset feature reconstruction mapping function to encode each point cloud modal feature to be processed respectively to obtain a fourth encoding result, and then using a second decoder to decode the fourth encoding result to obtain a corresponding image modal feature reconstruction result.

[0100] Furthermore, the bidirectional feature mapping framework includes four basic feature mapping modules, namely, the first mapping module , the second mapping module , the third mapping module and the fourth mapping module . The first mapping module and the second mapping module are used to map the source modality features to the intermediate features in the latent feature space, and the third mapping module and the fourth mapping module are used to map the intermediate features in the latent feature space to the target modality feature space. In a specific embodiment, the expression for predicting the intermediate features corresponding to the image modality features as the features corresponding to the point cloud modality is as follows:

[0101] ;

[0102] in, is the intermediate feature corresponding to the image modality feature.

[0103] The expression for predicting the intermediate features corresponding to the point cloud modal features as the features corresponding to the image modal features is as follows:

[0104] ;

[0105] in, It is the intermediate feature corresponding to the point cloud modal feature.

[0106] The expression for reconstructing the intermediate features corresponding to the point cloud modal features into the features corresponding to the point cloud modal features is as follows:

[0107] ;

[0108] The expression for reconstructing the intermediate features corresponding to the image modality features into the features corresponding to the image modality features is as follows:

[0109] ;

[0110] It is worth mentioning that the loss function of the bidirectional feature mapping framework is composed of the weighted sum of the loss corresponding to the cross-modal feature prediction task and the loss corresponding to the intra-modal feature reconstruction task. The calculation formula is as follows:

[0111] ;

[0112] Intra-modal feature reconstruction task loss The calculation formula is:

[0113] ;

[0114] Total loss function The calculation formula is:

[0115] ;

[0116] in, is the weight coefficient.

[0117] It is understandable that, from a structural perspective, the prediction task and reconstruction task of the same modality share the same encoder, enabling them to learn feature representations of the same object under different tasks. Accordingly, each decoder is responsible for both prediction and reconstruction tasks, and needs to accurately decode the latent features into target modality features. It is worth mentioning that under the dual supervision of the prediction task and the reconstruction task, the encoder and decoder of the shared component can realize the knowledge transfer and reuse between tasks, so that all modules can learn robust and universal latent features. Among them, the features not only need to retain the key information of the original modality, but also need to support accurate cross-modal conversion, thereby improving the accuracy of cross-modal prediction.

[0118] Step S14: determining an abnormality detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result, and the point cloud modal feature reconstruction result.

[0119] In a specific implementation, the embodiment of the present application uses cosine distance to calculate the prediction error of the cross-modal feature prediction task and the reconstruction error of the intra-modal feature reconstruction task, and multiplies the prediction error and the reconstruction error pixel by pixel to implement the logical AND operation between the errors to obtain the final pixel-by-pixel anomaly score map, and finally uses the maximum anomaly score in the anomaly score map as the anomaly score corresponding to the product to be detected.

[0120] It is worth mentioning that the acquisition of abnormal detection results of the product to be tested includes the offline model training stage and the online abnormal detection stage of the target product to be tested, such as Figure 3 As shown in the figure, the offline model training stage is to train the feature cross-modal prediction task and the intra-modal feature reconstruction task, and the online product anomaly detection stage is to calculate the anomaly score of the product to be detected by using the deviations corresponding to the feature cross-modal prediction and the intra-modal feature reconstruction task.

[0121] Furthermore, the training of the cross-modal prediction task and the intra-modal feature reconstruction task aims to learn the mapping relationship between the multi-modal features corresponding to the standard product, so it is necessary to use the multi-modal samples corresponding to the standard product for training. The training process is achieved by minimizing the weighted sum of the cross-modal prediction task loss and the intra-modal feature reconstruction task loss. Specifically, before using the preset cross-modal feature prediction task in the bidirectional feature mapping framework to perform cross-modal feature prediction on the image modal features to be processed and the point cloud modal features to be processed, it can also include: using the bidirectional feature mapping framework to process the image modalities and point cloud modalities corresponding to several standard products, respectively, to obtain corresponding third feature prediction results, fourth feature prediction results, first feature reconstruction results and second feature reconstruction results; determining a first loss result based on a similarity distance corresponding to the third feature prediction result and a similarity distance corresponding to the fourth feature prediction result; determining a second loss result based on a similarity distance corresponding to the first feature reconstruction result and a similarity distance corresponding to the second feature reconstruction result; setting a weight coefficient based on user needs, and determining a target loss result using the weight coefficient and based on the first loss result and the second loss result, so as to use the target loss result to adjust the parameters corresponding to the preset cross-modal feature prediction task and the preset intra-modal feature reconstruction task.

[0122] It is worth mentioning that the embodiment of the present application can use a preset optimizer and utilize the target loss result to adjust the parameters corresponding to the preset cross-modal feature prediction task and the preset intra-modal feature reconstruction task. In a specific implementation, the preset optimizer can be an Adam optimizer (Adam Optimize, an adaptive optimization algorithm), which is not specifically limited here.

[0123] It is understandable that the operation performed in the online target product anomaly detection stage is to obtain the corresponding anomaly score maps based on the first feature prediction result and the second feature prediction result corresponding to the cross-modal prediction deviation task and the image modal feature reconstruction result and the point cloud modal feature reconstruction result corresponding to the intra-modal feature reconstruction task. Among them, each anomaly score map characterizes the degree of anomaly of each pixel from the perspective of a single feature mapping task. It is understandable that different products and anomaly types usually show different patterns in different feature mapping tasks. Therefore, in this embodiment, an approximate multiplication logical AND operation is used to aggregate the anomaly scores of different anomaly score maps. Specifically, determining the abnormality detection result corresponding to the product to be inspected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result can include: determining the corresponding first abnormality score map, second abnormality score map, third abnormality score map and fourth abnormality score map based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result respectively; using a preset operation algorithm to operate the first abnormality score map, the second abnormality score map, the third abnormality score map and the fourth abnormality score map according to pixel points to obtain an abnormality score map to be processed; using a preset Gaussian smoothing processing technology to process the abnormality score map to be processed to obtain a target abnormality score map, and setting the abnormality score with the largest value in the target abnormality score map as the abnormality detection result corresponding to the product to be inspected.

[0124] In a specific implementation, the calculation formula of the abnormal score map is as follows:

[0125] ;

[0126] in, is the target anomaly score map, is the fourth abnormal score map, is the second anomaly score map, is the third anomaly score map, It is the first anomaly score map.

[0127] It is worth mentioning that the multiplication-approximated logical AND operation is able to filter out anomaly scores that are only significant in a single task, while retaining regions that are consistently identified as anomalies in all tasks.

[0128] Furthermore, before obtaining the target anomaly score map, the anomaly score map to be processed obtained after the operation needs to be Gaussian smoothed to obtain the target anomaly score map. Furthermore, the maximum anomaly score with the largest value in the target anomaly score map is the anomaly score corresponding to the product to be detected, which is used to determine whether the product to be detected has an anomaly.

[0129] It can be seen that before the multimodal industrial anomaly detection based on bidirectional feature mapping, the embodiment of the present application first needs to collect the image modality and point cloud modality corresponding to the product to be detected, and use the first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use the second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed. Subsequently, the preset cross-modal feature prediction task in the bidirectional feature mapping framework is used to perform cross-modal feature prediction on the image modality features to be processed and the point cloud modality features to be processed, and the corresponding first feature prediction results and second feature prediction results are obtained. Then, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the image modality features to be processed and the point cloud modality features to be processed, respectively, and the corresponding image modality feature reconstruction results and point cloud modality feature reconstruction results are obtained respectively. Finally, the abnormal detection result corresponding to the product to be detected is determined based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result. In this way, the accuracy and efficiency of abnormal detection of products are improved.

[0130] Accordingly, see Figure 4 As shown, the present application also provides a multimodal industrial anomaly detection device based on bidirectional feature mapping, comprising:

[0131] The feature extraction module 11 is used to collect the image modality and point cloud modality corresponding to the product to be inspected, and use the first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use the second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed;

[0132] A feature prediction module 12 is used to perform cross-modal feature prediction on the image modal features to be processed and the point cloud modal features to be processed based on a preset cross-modal feature prediction task in the bidirectional feature mapping framework to obtain a corresponding first feature prediction result and a second feature prediction result;

[0133] A feature reconstruction module 13 is used to reconstruct the image modal features to be processed and the point cloud modal features to be processed respectively based on the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework, and obtain corresponding image modal feature reconstruction results and point cloud modal feature reconstruction results respectively;

[0134] The detection result determination module 14 is used to determine the abnormality detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result and the point cloud modality feature reconstruction result.

[0135] As can be seen from the above, before the embodiment of the present application performs multimodal industrial anomaly detection based on bidirectional feature mapping, it is necessary to first collect the image modality and point cloud modality corresponding to the product to be detected, and use the first preset feature extractor to extract the two-dimensional features corresponding to the image modality to obtain the image modality features to be processed, and then use the second preset feature extractor to extract the three-dimensional features corresponding to the point cloud modality to obtain the corresponding point cloud modality features to be processed. Subsequently, the preset cross-modal feature prediction task in the bidirectional feature mapping framework is used to perform cross-modal feature prediction on the image modality features to be processed and the point cloud modality features to be processed, and the corresponding first feature prediction results and second feature prediction results are obtained. Then, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the image modality features to be processed and the point cloud modality features to be processed, respectively, and the corresponding image modality feature reconstruction results and point cloud modality feature reconstruction results are obtained respectively. Finally, the abnormal detection result corresponding to the product to be detected is determined based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result. In this way, the accuracy and efficiency of abnormal detection of products are improved.

[0136] In some specific implementations, the feature extraction module 11 may specifically include:

[0137] A modality acquisition unit, used to acquire image modalities and point cloud modalities corresponding to the product to be inspected; the image modalities and the point cloud modalities are mapped one-to-one on pixels;

[0138] a feature extraction subunit, configured to use a first preset feature extractor to divide the image modality into a plurality of image blocks of the same size according to a preset image division rule, and to perform feature extraction on each of the image blocks to obtain corresponding first feature information;

[0139] The first feature information processing unit is used to process each of the first feature information using a preset bilinear interpolation technology to obtain a modal feature of the image to be processed; the modal feature of the image to be processed is a two-dimensional feature.

[0140] In some specific implementations, the feature extraction module 11 may specifically include:

[0141] a point cloud modality processing unit, configured to process the point cloud modality using the second preset feature extractor and based on a preset sampling method to obtain a plurality of point groups;

[0142] The second feature information processing unit is used to extract features from each of the point groups to obtain corresponding second feature information, and process each of the second feature information using a preset point feature interpolation method to obtain a modal feature of the point cloud to be processed; the modal feature of the point cloud to be processed is a three-dimensional feature.

[0143] In some specific implementations, the feature prediction module 12 may specifically include:

[0144] A first feature result determination unit is used to use the first preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each of the image modal features to be processed to the corresponding point cloud modal features to be processed, so as to obtain a corresponding first feature prediction result;

[0145] Accordingly, in some specific implementations, the feature prediction module 12 may specifically include:

[0146] The second feature result determination unit is used to use the second preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features to obtain a corresponding second feature prediction result.

[0147] In some specific implementations, the feature prediction module 12 may specifically include:

[0148] A first encoding result decoding unit, used for encoding each of the to-be-processed image modal features to the corresponding to the to-be-processed point cloud modal features using the first encoder in the preset cross-modal feature prediction task to obtain a first encoding result, and then decoding the first encoding result using the first decoder in the preset cross-modal feature prediction task to obtain a corresponding first feature prediction result;

[0149] Accordingly, in some specific implementations, the feature prediction module 12 may specifically include:

[0150] A second encoding result decoding unit is used to encode each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features using the second encoder in the preset cross-modal feature prediction task to obtain a second encoding result, and then decode the second encoding result using the second decoder in the preset cross-modal feature prediction task to obtain a corresponding second feature prediction result;

[0151] Accordingly, in some specific implementations, the feature reconstruction module 13 may specifically include:

[0152] a third encoding result decoding unit, configured to respectively encode each of the to-be-processed image modality features using the first encoder and based on a first preset feature reconstruction mapping function to obtain a third encoding result, and then decode the third encoding result using the first decoder to obtain a corresponding image modality feature reconstruction result;

[0153] Accordingly, in some specific implementations, the feature reconstruction module 13 may specifically include:

[0154] The fourth encoding result decoding unit is used to use the second encoder and based on the second preset feature reconstruction mapping function to encode each of the to-be-processed point cloud modal features respectively to obtain a fourth encoding result, and then use the second decoder to decode the fourth encoding result to obtain a corresponding image modal feature reconstruction result.

[0155] In some specific implementations, the multimodal industrial anomaly detection device based on bidirectional feature mapping may further include:

[0156] A modality processing unit, used to process the image modalities and point cloud modalities corresponding to a number of standard products respectively using the bidirectional feature mapping framework to obtain corresponding third feature prediction results, fourth feature prediction results, first feature reconstruction results, and second feature reconstruction results;

[0157] A first loss result determining unit, configured to determine a first loss result based on a similarity distance corresponding to the third feature prediction result and a similarity distance corresponding to the fourth feature prediction result;

[0158] A second loss result determining unit, configured to determine a second loss result based on a similarity distance corresponding to the first feature reconstruction result and a similarity distance corresponding to the second feature reconstruction result;

[0159] A parameter adjustment unit is used to set a weight coefficient based on user needs, and use the weight coefficient to determine a target loss result based on the first loss result and the second loss result, so as to use the target loss result to adjust the parameters corresponding to the preset cross-modal feature prediction task and the preset intra-modal feature reconstruction task.

[0160] In some specific implementations, the detection result determination module 14 may specifically include:

[0161] an abnormal score map determining unit, configured to determine a corresponding first abnormal score map, a second abnormal score map, a third abnormal score map, and a fourth abnormal score map based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, respectively;

[0162] an abnormal score map operation unit, configured to use a preset operation algorithm to operate the first abnormal score map, the second abnormal score map, the third abnormal score map, and the fourth abnormal score map according to pixel points to obtain an abnormal score map to be processed;

[0163] The abnormality detection result determination unit is used to process the abnormality score map to be processed using a preset Gaussian smoothing processing technology to obtain a target abnormality score map, and set the abnormality score with the largest value in the target abnormality score map as the abnormality detection result corresponding to the product to be detected.

[0164] Furthermore, the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0167] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0168] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal industrial anomaly detection method based on bidirectional feature mapping, characterized in that: include: Collecting image modalities and point cloud modalities corresponding to the product to be inspected, and using a first preset feature extractor to extract two-dimensional features corresponding to the image modality to obtain image modal features to be processed, and then using a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed; Based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the image to be processed and the modal features of the point cloud to be processed to obtain corresponding first feature prediction results and second feature prediction results; Reconstructing the image modality features to be processed and the point cloud modality features to be processed respectively based on the preset intra-modality feature reconstruction task in the bidirectional feature mapping framework, and obtaining corresponding image modality feature reconstruction results and point cloud modality feature reconstruction results respectively; An abnormality detection result corresponding to the product to be detected is determined based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result.

2. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to claim 1 is characterized in that: The collecting of image modalities and point cloud modalities corresponding to the product to be inspected, and using a first preset feature extractor to extract two-dimensional features corresponding to the image modalities to obtain image modal features to be processed, includes: Collecting image modalities and point cloud modalities corresponding to the product to be inspected; the image modalities and the point cloud modalities are mapped one by one on pixels; Using a first preset feature extractor to divide the image modality into a number of image blocks of the same size according to a preset image division rule, and performing feature extraction on each of the image blocks to obtain corresponding first feature information; Each of the first feature information is processed using a preset bilinear interpolation technology to obtain a modal feature of the image to be processed; the modal feature of the image to be processed is a two-dimensional feature.

3. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to claim 1 is characterized in that: The step of extracting the three-dimensional features corresponding to the point cloud modality using the second preset feature extractor to obtain the corresponding point cloud modality features to be processed includes: Processing the point cloud modality using the second preset feature extractor and based on a preset sampling method to obtain a plurality of point groups; Feature extraction is performed on each of the point groups to obtain corresponding second feature information, and each of the second feature information is processed using a preset point feature interpolation method to obtain a point cloud modal feature to be processed; the point cloud modal feature to be processed is a three-dimensional feature.

4. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to claim 1 is characterized in that: Based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the image to be processed to obtain a corresponding first feature prediction result, including: Using the first preset feature prediction mapping function in the preset cross-modal feature prediction task, each of the image modal features to be processed is converted to the corresponding point cloud modal features to be processed for cross-modal feature prediction to obtain a corresponding first feature prediction result; Correspondingly, based on the preset cross-modal feature prediction task in the bidirectional feature mapping framework, cross-modal feature prediction is performed on the modal features of the point cloud to be processed to obtain the corresponding second feature prediction result, including: By utilizing the second preset feature prediction mapping function in the preset cross-modal feature prediction task, each of the to-be-processed point cloud modal features is converted to the corresponding to-be-processed image modal features for cross-modal feature prediction to obtain a corresponding second feature prediction result.

5. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to claim 4 is characterized in that: The using of the first preset feature prediction mapping function in the preset cross-modal feature prediction task to perform cross-modal feature prediction on each of the image modal features to be processed to the corresponding point cloud modal features to be processed to obtain a corresponding first feature prediction result includes: Using the first encoder in the preset cross-modal feature prediction task to encode each of the to-be-processed image modal features to the corresponding to the to-be-processed point cloud modal features to obtain a first encoding result, and then using the first decoder in the preset cross-modal feature prediction task to decode the first encoding result to obtain a corresponding first feature prediction result; Correspondingly, the second preset feature prediction mapping function in the preset cross-modal feature prediction task is used to perform cross-modal feature prediction on each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features to obtain a corresponding second feature prediction result, including: Using the second encoder in the preset cross-modal feature prediction task to encode each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features to obtain a second encoding result, and then using the second decoder in the preset cross-modal feature prediction task to decode the second encoding result to obtain a corresponding second feature prediction result; Correspondingly, the preset intra-modality feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modality features of the image to be processed to obtain the corresponding image modality feature reconstruction result, including: Using the first encoder and based on a first preset feature reconstruction mapping function, each of the image modality features to be processed is encoded respectively to obtain a third encoding result, and then using the first decoder to decode the third encoding result to obtain a corresponding image modality feature reconstruction result; Correspondingly, the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework is used to reconstruct the modal features of the point cloud to be processed, and the corresponding point cloud modal feature reconstruction result is obtained, including: The second encoder is used to encode each of the point cloud modal features to be processed based on a second preset feature reconstruction mapping function to obtain a fourth encoding result, and then the fourth encoding result is decoded by the second decoder to obtain a corresponding image modal feature reconstruction result.

6. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to claim 1 is characterized in that: Before performing cross-modal feature prediction on the to-be-processed image modal features and the to-be-processed point cloud modal features using the preset cross-modal feature prediction task in the bidirectional feature mapping framework, the method further includes: The image modalities and point cloud modalities corresponding to a number of standard products are processed using the bidirectional feature mapping framework to obtain corresponding third feature prediction results, fourth feature prediction results, first feature reconstruction results, and second feature reconstruction results; Determine a first loss result based on a similarity distance corresponding to the third feature prediction result and a similarity distance corresponding to the fourth feature prediction result; Determine a second loss result based on a similarity distance corresponding to the first feature reconstruction result and a similarity distance corresponding to the second feature reconstruction result; A weight coefficient is set based on user needs, and a target loss result is determined using the weight coefficient based on the first loss result and the second loss result, so as to use the target loss result to adjust the parameters corresponding to the preset cross-modal feature prediction task and the preset intra-modal feature reconstruction task.

7. The multimodal industrial anomaly detection method based on bidirectional feature mapping according to any one of claims 1 to 6, characterized in that: The determining, based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, the abnormality detection result corresponding to the product to be detected includes: Determine a corresponding first anomaly score map, a second anomaly score map, a third anomaly score map, and a fourth anomaly score map based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, respectively; Using a preset calculation algorithm to calculate the first abnormal score map, the second abnormal score map, the third abnormal score map, and the fourth abnormal score map according to pixel points to obtain an abnormal score map to be processed; The to-be-processed anomaly score map is processed using a preset Gaussian smoothing technique to obtain a target anomaly score map, and the anomaly score with the largest value in the target anomaly score map is set as the anomaly detection result corresponding to the to-be-detected product.

8. A multimodal industrial anomaly detection device based on bidirectional feature mapping, characterized in that: include: A feature extraction module, used to collect image modalities and point cloud modalities corresponding to the product to be inspected, and use a first preset feature extractor to extract two-dimensional features corresponding to the image modality to obtain image modal features to be processed, and then use a second preset feature extractor to extract three-dimensional features corresponding to the point cloud modality to obtain corresponding point cloud modal features to be processed; A feature prediction module, used to perform cross-modal feature prediction on the image modality feature to be processed and the point cloud modality feature to be processed based on a preset cross-modal feature prediction task in the bidirectional feature mapping framework, and obtain a corresponding first feature prediction result and a second feature prediction result; A feature reconstruction module, used to reconstruct the image modal features to be processed and the point cloud modal features to be processed respectively based on the preset intra-modal feature reconstruction task in the bidirectional feature mapping framework, and obtain corresponding image modal feature reconstruction results and point cloud modal feature reconstruction results respectively; A detection result determination module is used to determine an abnormality detection result corresponding to the product to be detected based on the first feature prediction result, the second feature prediction result, the image modal feature reconstruction result and the point cloud modal feature reconstruction result.

9. The multimodal industrial anomaly detection device based on bidirectional feature mapping according to claim 8 is characterized in that: The feature prediction module comprises: A first encoding result decoding unit, used for encoding each of the to-be-processed image modal features to the corresponding to the to-be-processed point cloud modal features using the first encoder in the preset cross-modal feature prediction task to obtain a first encoding result, and then decoding the first encoding result using the first decoder in the preset cross-modal feature prediction task to obtain a corresponding first feature prediction result; Accordingly, the feature prediction module includes: A second encoding result decoding unit is used to encode each of the to-be-processed point cloud modal features to the corresponding to the to-be-processed image modal features using the second encoder in the preset cross-modal feature prediction task to obtain a second encoding result, and then decode the second encoding result using the second decoder in the preset cross-modal feature prediction task to obtain a corresponding second feature prediction result; Accordingly, the feature reconstruction module includes: a third encoding result decoding unit, configured to respectively encode each of the to-be-processed image modality features using the first encoder and based on a first preset feature reconstruction mapping function to obtain a third encoding result, and then decode the third encoding result using the first decoder to obtain a corresponding image modality feature reconstruction result; Accordingly, the feature reconstruction module includes: The fourth encoding result decoding unit is used to use the second encoder and based on the second preset feature reconstruction mapping function to encode each of the to-be-processed point cloud modal features respectively to obtain a fourth encoding result, and then use the second decoder to decode the fourth encoding result to obtain a corresponding image modal feature reconstruction result.

10. The multimodal industrial anomaly detection device based on bidirectional feature mapping according to claim 8 is characterized in that: The detection result determination module comprises: an abnormal score map determining unit, configured to determine a corresponding first abnormal score map, a second abnormal score map, a third abnormal score map, and a fourth abnormal score map based on the first feature prediction result, the second feature prediction result, the image modality feature reconstruction result, and the point cloud modality feature reconstruction result, respectively; an abnormal score map operation unit, configured to use a preset operation algorithm to operate the first abnormal score map, the second abnormal score map, the third abnormal score map, and the fourth abnormal score map according to pixel points to obtain an abnormal score map to be processed; The abnormality detection result determination unit is used to process the abnormality score map to be processed using a preset Gaussian smoothing processing technology to obtain a target abnormality score map, and set the abnormality score with the largest value in the target abnormality score map as the abnormality detection result corresponding to the product to be detected.

Images