Industrial anomaly detection method and equipment

Abstract

The invention provides an industrial anomaly detection method and equipment. The method comprises the following steps: acquiring multi-modal data, wherein the multi-modal data comprises a two-dimensional image feature and a three-dimensional point cloud feature; inputting the two-dimensional image features and the three-dimensional point cloud features into a trained unsupervised feature fusion model, and enabling the unsupervised feature fusion model to output target features; wherein the unsupervised feature fusion model is used for performing feature extraction and fusion on two-dimensional image features and three-dimensional point cloud features; and inputting the target feature into a preset decision model, and processing to obtain a detection result of the industrial product. The method is used for achieving the effect of improving the industrial anomaly detection precision.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an industrial anomaly detection method and device. Background Technology

[0002] In the industrial manufacturing sector, industrial product quality inspection is a key link in ensuring production safety and efficiency and reducing maintenance costs.

[0003] Currently, two-dimensional anomaly detection methods in the industrial field are mainly divided into two categories: reconstruction-based methods and pre-trained feature extractor-based methods. The former learns the representation of normal samples through image reconstruction tasks and uses this as the benchmark for anomaly detection; the latter uses pre-trained deep models to extract features, maps the features to a specific distribution space, and identifies samples that deviate from the normal distribution as anomalies. For the processing of multimodal data, different modal features are directly connected.

[0004] However, when the above method is used to process high-dimensional features, it can easily lead to mutual interference between features, reducing detection accuracy. Summary of the Invention

[0005] This application provides an industrial anomaly detection method and equipment to solve the problem of low accuracy in industrial product anomaly detection in the prior art.

[0006] In a first aspect, this application provides an industrial anomaly detection method, comprising:

[0007] Acquire multimodal data, which includes two-dimensional image features and three-dimensional point cloud features;

[0008] Two-dimensional image features and three-dimensional point cloud features are input into a trained unsupervised feature fusion model, which outputs the target features. The unsupervised feature fusion model is used to extract and fuse features from two-dimensional image features and three-dimensional point cloud features.

[0009] The target features are input into a pre-defined decision model, and the results of the industrial product testing are obtained.

[0010] In some embodiments, two-dimensional image features and three-dimensional point cloud features are input into a trained unsupervised feature fusion model, causing the unsupervised feature fusion model to output target features, including:

[0011] Align the 3D point cloud features with the 2D image features to obtain the 2D point cloud features;

[0012] Two-dimensional image features and two-dimensional point cloud features are input into a trained unsupervised feature fusion model, which then outputs the target features.

[0013] In some embodiments, aligning three-dimensional point cloud features with two-dimensional image features to obtain two-dimensional point cloud features includes:

[0014] The farthest point sampling algorithm is used to process the 3D point cloud features to obtain multiple center point features;

[0015] Interpolation calculations are performed on the center point features in the 3D point cloud features to obtain the 3D center point features corresponding to the center point features;

[0016] The three-dimensional center point features are projected onto a two-dimensional plane to obtain two-dimensional point cloud features.

[0017] In some embodiments, the farthest point sampling algorithm is used to process the 3D point cloud features to obtain multiple center point features, including:

[0018] The farthest point sampling algorithm is used to sample the 3D point cloud features to obtain N target point cloud features, where N is a positive integer;

[0019] Based on the target point cloud features, the 3D point cloud features are grouped to obtain N groups of point cloud features;

[0020] Each set of point cloud features is encoded into a feature vector and input into a preset feature extraction model, so that the feature extraction model outputs the center point features corresponding to each set of point cloud features.

[0021] In some embodiments, the three-dimensional center point features are projected onto a two-dimensional plane to obtain two-dimensional point cloud features, including:

[0022] Using preset camera parameters, the 3D center point features are projected onto a 2D plane to obtain the 2D center point features corresponding to the 3D center point features.

[0023] If the two-dimensional center point feature matches the feature point in the two-dimensional image feature, then the two-dimensional center point feature is retained;

[0024] If the two-dimensional center point feature does not match the feature points in the two-dimensional image feature, then the two-dimensional center point feature is set to 0;

[0025] Average pooling is used to fill in the empty positions of the two-dimensional center point features to obtain two-dimensional point cloud features.

[0026] In some embodiments, the target features are input into a preset decision model, and the resulting industrial product test results are obtained, including:

[0027] The target features are input into a preset decision model, which processes the target features to obtain an industrial product anomaly score and industrial product anomaly area information. The industrial product anomaly score is used to characterize the probability of anomalies in industrial products, and the industrial product anomaly area information is used to characterize the area with the highest probability of anomalies in industrial products.

[0028] In some embodiments, the multimodal data further includes sensor data; the method further includes:

[0029] Sensor data is input into a preset decision model, which then processes the sensor data to obtain information on abnormal operating status of industrial equipment.

[0030] In some embodiments, the method further includes:

[0031] The unsupervised feature fusion model is trained using a patch-contrast loss function, and the learning rate is adjusted using a cosine function to obtain the trained unsupervised feature fusion model.

[0032] Secondly, this application provides an industrial anomaly detection device, comprising:

[0033] The acquisition module is used to acquire multimodal data, which includes two-dimensional image features and three-dimensional point cloud features.

[0034] The processing module is used to input two-dimensional image features and three-dimensional point cloud features into the trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs the target features; wherein, the unsupervised feature fusion model is used to extract and fuse features from two-dimensional image features and three-dimensional point cloud features;

[0035] The processing module is also used to input the target features into a preset decision model and process them to obtain the detection results of industrial products.

[0036] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0037] The memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory to implement the industrial anomaly detection method of the first aspect and any embodiment of the first aspect.

[0038] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the industrial anomaly detection method of the first aspect and any embodiment of the first aspect.

[0039] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the industrial anomaly detection method of the first aspect and any embodiment of the first aspect.

[0040] The industrial anomaly detection method and equipment provided in this application use an unsupervised feature fusion model to process multimodal data, obtain target features, and use a decision model to detect anomalies in the target features, thereby improving the accuracy of industrial anomaly detection, reducing detection labor costs, and increasing detection efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating an industrial anomaly detection method provided in one embodiment of this application;

[0043] Figure 2 This is a partial flowchart of an industrial anomaly detection method provided in one embodiment of this application;

[0044] Figure 3 A schematic flowchart of an industrial anomaly detection method provided in an embodiment of this application;

[0045] Figure 4 An application example diagram of an industrial anomaly detection method provided in an embodiment of this application is shown.

[0046] Figure 5 This is a schematic diagram of the structure of an industrial anomaly detection device provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Depending on the context, the word “if” as used in this application can be interpreted as “when”, “when”, or “in response to determination”.

[0050] Quality inspection of industrial products and equipment is crucial for preventing safety accidents, improving production efficiency, and reducing maintenance costs. Traditional equipment anomaly detection methods typically rely solely on image data to detect malfunctions. However, image data may not capture all the characteristics of equipment failures, as factors such as noise, lighting conditions, and viewing angle can affect accuracy. In industrial product quality inspection, the quality of most components depends on the visual inspection and experience of personnel. Inspectors usually use three-dimensional shape and color features to determine the presence of defects, with three-dimensional shape information being critical for accurate detection. However, prolonged work can lead to missed or false detections due to visual fatigue, lack of concentration, the variety of product defect types, and high inspection requirements, thus reducing production efficiency. Furthermore, manual quality inspection requires handling products, which can damage them during transport and lead to quality complaints.

[0051] Currently, two-dimensional industrial anomaly detection methods can be divided into two categories. One is a reconstruction-based method, which learns the representation of normal samples through image reconstruction tasks and uses this as the benchmark for anomaly detection. This method is easy to reconstruct for single-modal inputs, such as two-dimensional images or three-dimensional point clouds, but it is difficult to find the reconstruction target for multi-modal inputs. The other is a method based on pre-trained feature extractors, which maps the extracted features to a normal distribution map as a normal representation and identifies features that deviate from the normal distribution as anomalies.

[0052] For multimodal industrial anomaly detection methods, the features of the two modes mentioned above are usually directly connected together. However, when the feature dimension is high, the multimodal features will interfere with each other, resulting in a decrease in detection performance.

[0053] To address the aforementioned issues, this application proposes an industrial anomaly detection method and device. This method employs an unsupervised feature fusion model to integrate multimodal features, and a decision model to detect anomalies in both the multimodal features and the fused features. This improves the accuracy of 3D industrial inspection, reduces labor costs, and increases inspection efficiency.

[0054] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] In this application, an electronic device is used as the execution subject to perform the industrial anomaly detection method of the following embodiments. Specifically, the execution subject can be the hardware device of the electronic device, or the software application implementing the following embodiments in the electronic device, or a computer-readable storage medium on which the software application implementing the following embodiments is installed, or the code implementing the software application of the following embodiments.

[0056] Figure 1 A flowchart of an industrial anomaly detection method according to an embodiment of this application is shown. Figure 1 As shown, with an electronic device as the execution subject, the method in this embodiment may include the following steps:

[0057] S101. Obtain multimodal data.

[0058] In this embodiment, the multimodal data includes two-dimensional image features and three-dimensional point cloud features.

[0059] Specifically, when electronic devices acquire multimodal data, they can acquire two-dimensional images and three-dimensional scan data of industrial products. The two-dimensional images and three-dimensional scan data are used to characterize the appearance and shape information of industrial products.

[0060] Electronic devices can remove backgrounds from 2D images and 3D scan data, thereby generating 2D image features from the background-removed 2D image and 3D point cloud features from the background-removed 3D scan data. These 2D image features and 3D point cloud features can characterize key attributes of industrial products, such as their appearance texture, color distribution, and 3D shape.

[0061] Optionally, the electronic device can employ a pre-trained feature extractor to extract two-dimensional image features from two-dimensional images and three-dimensional point cloud features from three-dimensional scan data. For example, the DETR with Improved DeNoising Anchor Boxes (DINO) feature extractor can be used to extract two-dimensional image features from RGB (Red, Green, Blue) images, and a Masked Auto-encoders for Point Cloud Understanding (PointMAE) deep learning model specifically designed for point cloud processing can be used to extract three-dimensional point cloud features from three-dimensional scan data.

[0062] S102. Input the two-dimensional image features and three-dimensional point cloud features into the trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs the target features.

[0063] Specifically, since the interaction between multimodal features can generate new representations that help with anomaly detection, this embodiment uses an unsupervised feature fusion model to extract and fuse two-dimensional image features and three-dimensional point cloud features, thereby unifying the distribution of multimodal features and learning the intrinsic relationships between multimodal features.

[0064] Unsupervised feature fusion models, based on deep learning architectures, automatically learn the correlations and complementarities between different features, thereby generating a target feature that contains richer and more comprehensive information. This process is unsupervised, meaning that no additional labeled data is required to guide the fusion process.

[0065] S103. Input the target features into the preset decision model and process them to obtain the detection results of industrial products.

[0066] In this embodiment, an unsupervised feature fusion model is used to process multimodal data, which helps improve the performance of industrial anomaly detection. However, it inevitably leads to the loss of information in each single domain. Therefore, this embodiment also employs a decision model to analyze and process the target features obtained from the unsupervised feature fusion model, ultimately yielding the detection results of the industrial product. The detection results may include information such as whether the product is qualified, the specific type of defect, and the location of the defect.

[0067] Electronic devices can also display the detection results to users in a visual form, such as images on a screen, indicator light status, or sound prompts, and store the detection results in internal memory or transmit them to a remote server via a network for further analysis or recording.

[0068] The industrial anomaly detection method provided in this embodiment uses an unsupervised feature fusion model and a decision model to process multimodal data, reducing interference between multimodal data and encouraging feature interaction, thereby better perceiving the complexity of industrial anomalies, improving the accuracy of anomaly detection, reducing detection labor costs, and improving detection efficiency.

[0069] In some embodiments, step S102 is specifically implemented as follows:

[0070] S201. Align the 3D point cloud features with the 2D image features to obtain the 2D point cloud features.

[0071] S202. Input the two-dimensional image features and two-dimensional point cloud features into the trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs the target features.

[0072] Specifically, the electronic device aligns 3D point cloud features with 2D image features, aiming to map points in 3D space to pixels in the 2D image, thereby obtaining 2D point cloud features—that is, converting point group features into planar features. The electronic device then inputs the 2D image features and 2D point cloud features into a trained unsupervised feature fusion model. The unsupervised feature fusion model extracts color features from the 2D image features, fuses the color features and 2D point cloud features to obtain fused features, and finally combines the color features, 2D point cloud features, and fused features to output the target features.

[0073] In this embodiment, the electronic device obtains target features based on 3D point cloud features and 2D image features using a trained unsupervised feature fusion model. This promotes cross-domain information interaction while preserving the original information of each domain. It also solves the problem of mismatch between the positional information of 2D image features and 3D point cloud features, aligns 3D point cloud features with 2D image features, simplifies multimodal interaction, and improves the accuracy of anomaly detection.

[0074] In some embodiments, step S201 is specifically implemented as follows:

[0075] S301. Using the farthest point sampling algorithm, the 3D point cloud features are processed to obtain multiple center point features.

[0076] The electronic device can randomly select an initial point as the first center point feature, or it can use a point in the dataset that has representative or special properties as the first center point feature.

[0077] The electronic device calculates the distance from each point in the dataset to the nearest point in the currently selected center point feature set. It selects the point with the largest distance as the next center point feature and adds it to the center point feature set, thus obtaining multiple center point features. This process is repeated until a preset number of center points is reached or other stopping conditions are met.

[0078] S302. In the three-dimensional point cloud features, perform interpolation calculation on the center point features to obtain the three-dimensional center point features corresponding to the center point features.

[0079] The electronic device can determine the region and range for difference calculation based on multiple center point features, and select an appropriate interpolation method, such as linear interpolation, radial basis function interpolation, or moving least squares interpolation. Within the selected interpolation region, the electronic device performs interpolation calculations for each center point feature to obtain its corresponding three-dimensional center point feature.

[0080] Electronic devices can verify the interpolation results to ensure that the interpolated 3D center point features are consistent and coherent with the original 3D point cloud features in terms of geometry and color.

[0081] S303. Project the three-dimensional center point features onto a two-dimensional plane to obtain two-dimensional point cloud features.

[0082] The electronic device can select a suitable projection method, such as orthographic projection or perspective projection. It transforms the coordinates of each 3D center point feature onto a selected 2D plane. During the projection process, the electronic device retains as much effective information as possible from the 3D center point features, such as color and texture, ultimately outputting the projected 2D point cloud features.

[0083] In some embodiments, step S301 is specifically implemented as follows:

[0084] S3011. Using the farthest point sampling algorithm, sample the three-dimensional point cloud features to obtain N target point cloud features.

[0085] Where N is a positive integer.

[0086] Specifically, the electronic device can randomly select a point from the dataset as the starting point, that is, the first target point cloud feature.

[0087] The electronic device calculates the distance from the remaining points in the dataset to the nearest point in the currently selected target point cloud feature set. It selects the point with the largest distance as the next target point cloud feature and adds it to the target point cloud feature set. This process is repeated until N target point cloud features are selected or other stopping conditions are met, such as reaching a preset distance threshold or a time limit.

[0088] S3012. Based on the target point cloud features, group the three-dimensional point cloud features to obtain N groups of point cloud features.

[0089] The electronic device groups the original 3D point cloud features based on the proximity of the 3D point cloud features to the target point cloud features. Specifically, it can choose grouping strategies such as nearest neighbor search or K-means clustering. This embodiment does not limit this.

[0090] In this embodiment, each target point cloud feature corresponds to a set of point cloud features, which are spatially close to the target point cloud feature. For each target point cloud feature, the electronic device searches for points in the original point cloud features whose distance is less than a certain threshold and groups these points into the same group. Alternatively, the electronic device uses a clustering algorithm to cluster the point cloud features, with each cluster center corresponding to a target point cloud feature. Point cloud features within the same cluster are grouped into the same group, thus obtaining N sets of point cloud features.

[0091] S3013. Encode each set of point cloud features into a feature vector and input it into a preset feature extraction model so that the feature extraction model outputs the center point features corresponding to each set of point cloud features.

[0092] The electronic device encodes each set of point cloud features, converting them into a fixed-length feature vector. This can be achieved through statistical features, local feature extraction, or deep learning methods.

[0093] Electronic devices can process input feature vectors using a pre-defined feature extraction model to obtain the center point features corresponding to each set of point cloud features. The pre-defined feature extraction model can be a pre-trained deep learning network model, such as a convolutional neural network or a graph neural network.

[0094] Figure 2 This is a partial flowchart of an industrial anomaly detection method provided in one embodiment of this application. Figure 2 As shown, in some embodiments, step S303 is specifically implemented as follows:

[0095] S3031. Using preset camera parameters, project the three-dimensional center point features onto a two-dimensional plane to obtain the two-dimensional center point features corresponding to the three-dimensional center point features.

[0096] Specifically, the electronic device acquires preset camera parameters, including intrinsic and extrinsic parameters. Then, using the principle of perspective projection and in conjunction with the camera parameters, it projects each 3D center point feature from 3D space onto a 2D plane, obtaining a set of 2D center point features corresponding to the 3D center point features. These features are represented by pixel coordinates on the 2D plane.

[0097] S3032. If the two-dimensional center point feature matches the feature point in the two-dimensional image feature, then the two-dimensional center point feature is retained.

[0098] S3033. If the two-dimensional center point feature does not match the feature points in the two-dimensional image feature, then set the two-dimensional center point feature to 0.

[0099] In this embodiment, for each two-dimensional center point feature, the electronic device determines whether it successfully matches a corresponding feature point in the two-dimensional image. Specifically, if the matching similarity is greater than a threshold, the match is considered successful, and the two-dimensional center point feature is retained. If the matching similarity is less than or equal to the threshold, the match is unsuccessful, and the value of the two-dimensional center point feature is set to 0, indicating that these points do not have corresponding feature points in the two-dimensional image.

[0100] S3034. Average pooling is used to fill in the empty positions of the two-dimensional center point features to obtain two-dimensional point cloud features.

[0101] In this embodiment, the electronic device first defines a pooling region, which is a fixed-size window used to perform local averaging on the two-dimensional center point features. Within the pooling region, the electronic device calculates the mean of the non-zero two-dimensional center point features and assigns this mean to all two-dimensional center point features with a value of 0 within the pooling region, thereby smoothing and supplementing missing feature information.

[0102] After average pooling, the electronic device can obtain a complete set of two-dimensional point cloud features. These features are represented by dense pixels on a two-dimensional plane and can be used for subsequent image processing and analysis tasks.

[0103] In some embodiments, step S103 is specifically implemented by: inputting the target features into a preset decision model, so that the decision model processes the target features to obtain industrial product anomaly scores and industrial product anomaly area information.

[0104] Among them, the industrial product anomaly score is used to characterize the probability of anomalies in industrial products, and the industrial product anomaly area information is used to characterize the area with the highest probability of anomalies in industrial products.

[0105] Optionally, the decision model may include a scoring module and an image analysis module. The scoring module processes target features to obtain anomaly scores for the product; the image analysis module processes target features to obtain abnormal regions of the product.

[0106] In some embodiments, the multimodal data also includes sensor data.

[0107] Optionally, the electronic device can also input sensor data into a preset decision model, so that the decision model can process the sensor data and obtain abnormal information about the operating status of the industrial equipment.

[0108] Electronic devices acquire sensor data and process it using decision models, enabling intelligent monitoring and anomaly diagnosis of industrial equipment operation status. By using the hybrid fusion of multimodal data, the complexity of industrial anomalies can be better captured, thereby improving the accuracy of anomaly detection.

[0109] Specifically, in industrial quality inspection, image data can be used to detect external anomalies in machines, such as deformation, oil stains, and wear. Point cloud data can be used to detect internal anomalies, such as bearing wear and gear wear. Sensor data can be used to detect the machine's operating status, such as temperature, vibration, and noise.

[0110] The industrial anomaly detection method in this embodiment can fuse features from different data sources to obtain more comprehensive and accurate anomaly detection results, thereby improving the reliability and safety of equipment and reducing maintenance costs.

[0111] In some embodiments, the electronic device trains the unsupervised feature fusion model using a patch contrastive loss function and adjusts the learning rate using a cosine function to obtain a trained unsupervised feature fusion model.

[0112] Figure 3 This is a schematic flowchart illustrating an industrial anomaly detection method according to an embodiment of this application. Figure 3 As shown, using electronic devices as the executing entity, the industrial anomaly detection method of this embodiment includes the following steps:

[0113] S401, Point Feature Alignment.

[0114] To address the mismatch between the positional information of 2D image features and 3D point cloud features, this embodiment aligns 3D features with 2D space through point feature alignment, simplifying multimodal interaction and improving detection performance.

[0115] Specifically, point feature alignment transforms point group features into planar features through interpolation and projection operations, where FPS stands for Farthest Point Sampling and Fpt stands for Pre-trained Point Transformer.

[0116] Point feature extraction specifically involves using Fpt to extract 3D point cloud features. The input point cloud p represents a sequence of point positions with N points. After farthest point sampling (FPS), the point cloud is divided into multiple groups, each containing multiple points. Then, the points in each group are encoded into a feature vector, and this feature vector is input into Fpt, which outputs the point features. The point features are divided into multiple feature groups, each with a single-point feature, which can be considered as the feature of the center point.

[0117] After sampling the farthest point, the center points are unevenly distributed in space, resulting in an imbalance in point feature density. Therefore, the features are interpolated back to the original point cloud. Given a point feature, it is associated with the center point, and the inverse distance weights are used to interpolate the feature to each point in the input point cloud p.

[0118] After interpolation, the 3D center point features are projected onto a 2D plane using point coordinates and camera parameters to obtain the 2D center point features corresponding to the 3D center point features. If the 2D center point features do not match the feature points in the 2D image features, the 2D center point features are set to 0. Then, average pooling is used to fill in the empty positions of the 2D center point features to obtain the 2D point cloud features.

[0119] S402, Unsupervised feature fusion.

[0120] Since the interactions between multimodal features can generate new representations that help with anomaly detection, this embodiment uses an unsupervised feature fusion model to unify the distribution of multimodal features and learn the intrinsic relationships between them.

[0121] Unsupervised feature fusion specifically involves comparing point features and image features block by block with the loss function Lcon, thereby learning the inherent relationships between multimodal feature blocks at the same location.

[0122] Frgb stands for Vision Transformer, which is used to extract RGB features. and For a multi-layer perceptron (MLP) layer, and It is a single fully connected layer.

[0123] Interactions between cross-modal features can generate new information, which is of great significance for industrial anomaly detection. To learn the intrinsic relationships between different modalities in the training data, this embodiment uses a patch contrastive loss function to train the feature fusion module. Specifically, based on given RGB feature Frgb and point cloud feature Fpt, it enhances the correlation of features at the same location and reduces the correlation of features at different locations.

[0124] S403, Integration of Decision-Making Level.

[0125] Using unsupervised feature fusion can help improve detection performance, but it inevitably loses information from each individual domain. Therefore, this embodiment uses decision-level fusion to make the final decision using multiple repositories.

[0126] Among them, Mrgb, Mfs, and Mpt are three repositories used to store original color features, location features, and fused features, respectively. Da and Ds are two learnable modules that can be used to output industrial product anomaly scores and industrial product anomaly area information, respectively. , It is a score function for a single memory repository detection, and P is the repository construction algorithm.

[0127] The decision-level fusion combines multimodal data with multiple repositories and uses Da and Ds to make the final anomaly detection decision.

[0128] It should be understood that industrial anomalies typically occur only in a single domain, and the correspondence between cross-modal features may not be obvious. Furthermore, while feature fusion facilitates interaction between multimodal features, some information may be lost during the fusion process. Therefore, this embodiment employs multiple repositories to store the original color features, location features, and fused features. During decision-making, each repository is used to predict anomaly scores, and two learnable One-Class Support Vector Machines (OCSVMs) Da and Ds are used for the final evaluation of the anomaly scores.

[0129] Figure 4 This diagram illustrates an application example of an industrial anomaly detection method provided in an embodiment of this application. (See diagram for example.) Figure 4 As shown, with an electronic device as the execution subject, the method in this embodiment includes the following steps:

[0130] S501, Acquire multimodal data.

[0131] Multimodal data includes images, point clouds, and sensor data.

[0132] The MVTec 3D Anomaly Detection Dataset (MVTec-3DAD) is the first 3D industrial anomaly detection dataset. The method in this embodiment is performed on the MVTec-3DAD dataset.

[0133] The MVTec-3D AD dataset comprises 10 classes, a total of 2656 training samples, and 1137 test samples. 3D scans were acquired using structured light from industrial sensors, and positional information is stored in a 3-channel tensor representing x, y, and z coordinates. These 3-channel tensors can be individually mapped to corresponding point clouds. Furthermore, RGB information is recorded for each point; since all samples in the dataset are viewed from the same angle, the RGB information for each sample can be stored in a single image. Each sample in the MVTec-3D AD dataset contains a color point cloud.

[0134] S502. Preprocess the multimodal data.

[0135] Preprocessing includes feature extraction and feature dimensionality reduction.

[0136] Unlike 2D data, 3D data is easier to remove background information from. This embodiment uses the RANSAC (RANdomSample Consensus) method to estimate the background plane, and any point within 0.005 of the background plane is removed. Simultaneously, the corresponding pixels of the removed points in the RGB image are set to 0. This operation not only accelerates 3D feature processing during training and inference but also reduces the interference of the background on anomaly detection. Finally, the size of both the position tensor and the RGB image is adjusted to 224×224, consistent with the input size of the feature extractor.

[0137] For feature extraction, this embodiment uses the ViT-B / 8 architecture's VisionTransformer for RGB features to suit the efficiency and granularity of anomaly detection tasks. To achieve higher performance, a ViT-B / 8 model pre-trained on ImageNet using DINO is used. This pre-trained model can receive 224×224 images and output 784 supplementary features for each image. ViT focuses on both global and local information in the preceding and subsequent layers; therefore, the output of the final layer with 768 dimensions is used for anomaly detection.

[0138] For point cloud features, this embodiment uses a Point Transformer pre-trained on the ShapeNet dataset as a 3D feature extractor, and uses the output of layers {3, 7, 11} as 3D features. The Point Transformer first encodes the point cloud into point groups similar to ViT supplementation, with each group having a center point as its location and the group size as its neighbor count. During implementation, settings M=784, S=64 and M=1024, S=128 were tested. In the PFA operation, point features were pooled to 28×28 and 56×56 for testing.

[0139] S503. Use multimodal data to train an anomaly detection model.

[0140] The anomaly detection model includes an unsupervised feature fusion module and a decision-level fusion module.

[0141] For unsupervised feature fusion , It consists of two two-layer MLPs, each with 4 hidden dimensions for each input feature. Using the AdamW optimizer, the learning rate is set to 0.003, and the learning rate is gradually increased over 250 steps with cosine forward feedback. The learning rate is of size 256, and the best anomaly detection result is output after 750 unsupervised feature fusion training steps.

[0142] For decision-level fusion, two linear OCSVMs with stochastic gradient descent (SGD) optimizers are used, with a learning rate set to 1×10⁻⁶. -4 And train 1000 steps for each category.

[0143] S504, Define evaluation metrics and test the anomaly detection model.

[0144] In this embodiment, the Image Level Area Under the Receiver Operating Characteristic Curve (I-AUROC) is used to evaluate the image-level anomaly detection performance. A higher I-AUROC indicates better image-level anomaly detection performance. The MVTec-3D AD dataset test set is input into the model for evaluation, and the corresponding anomaly image annotations are output along with the evaluation metrics.

[0145] In this embodiment of the industrial anomaly detection method, to maintain the single-domain inference capability of the anomaly detection model, repositories are constructed for RGB, 3D features, and fused features. Given that anomaly detection requires features that simultaneously include global and local information—where local information helps detect small defects and global information focuses on the relationship between global and local information—this embodiment uses a Point Transformer to extract 3D features and a Vision Transformer to extract RGB features. Then, a point feature alignment method is employed to align the 3D and 2D features, ensuring consistent detection results.

[0146] Unlike the approach using a unimodal model like Asymmetric student-teacher networks for industrial anomaly detection (AST), the method in this embodiment achieves superior results on an RGB+3D multimodal anomaly detection dataset under the I-AUROC standard, improving the accuracy by 0.8% compared to the original AST approach, from 0.937 to 0.945. Therefore, this embodiment's method, as a hybrid fusion scheme, utilizes point feature alignment, unsupervised feature fusion, and decision-level fusion to process multimodal datasets, learning the inherent relationships between multimodal feature blocks at the same location. This allows for a better perception of the complexity of industrial anomalies, improving the accuracy of industrial anomaly detection, effectively reducing the cost of inspection personnel, increasing detection efficiency, and covering various industrial anomaly detection scenarios, including machine monitoring, product quality inspection, and safety early warning.

[0147] Figure 5 This application provides a schematic diagram of the structure of an industrial anomaly detection device according to an embodiment of the present application. Figure 5 As shown, the industrial anomaly detection device 10 of this embodiment is used to implement the operation corresponding to the electronic device in any of the above method embodiments. The industrial anomaly detection device 10 of this embodiment includes:

[0148] The acquisition module 11 is used to acquire multimodal data, which includes two-dimensional image features and three-dimensional point cloud features;

[0149] Processing module 12 is used to input two-dimensional image features and three-dimensional point cloud features into a trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs target features; wherein, the unsupervised feature fusion model is used to extract and fuse features from two-dimensional image features and three-dimensional point cloud features;

[0150] The processing module 12 is also used to input the target features into a preset decision model and process them to obtain the detection results of industrial products.

[0151] The industrial anomaly detection device 10 provided in this embodiment can execute the above method embodiment. Its specific implementation principle and technical effect can be found in the above method embodiment, and will not be repeated here.

[0152] Figure 6 A schematic diagram of the hardware structure of an electronic device according to an embodiment of this application is shown. Figure 6 As shown, the electronic device 20 is used to implement the operation corresponding to the electronic device in any of the above method embodiments. The electronic device 20 in this embodiment may include: a memory 21, a processor 22, and a communication interface 24.

[0153] The memory 21 is used to store computer programs. The memory 21 may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0154] Processor 22 is used to execute computer programs stored in memory to implement the methods in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments. The processor 22 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0155] Alternatively, the memory 21 can be either standalone or integrated with the processor 22.

[0156] When the memory 21 is a device independent of the processor 22, the electronic device 20 may also include a bus 23. This bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0157] The communication interface 24 can be connected to the processor 22 via the bus 23. The processor 22 can control the communication interface 24 to realize the functions of receiving and sending signals.

[0158] The electronic device 20 provided in this embodiment can be used to perform the above-described industrial anomaly detection method. Its implementation method and technical effect are similar, and will not be described again in this embodiment.

[0159] This application also provides a computer-readable storage medium storing a computer program / instructions, which, when executed by a processor, are used to implement the methods provided in the various embodiments described above.

[0160] The computer-readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of a computer program from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the ASIC can reside in a user equipment. Of course, the processor and the computer-readable storage medium can also exist as discrete components in a communication device.

[0161] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0162] This application also provides a computer program product comprising a computer program / instructions stored in a computer-readable storage medium. At least one processor of the device can read the computer program / instructions from the computer-readable storage medium, and the at least one processor executes the computer program / instructions to cause the device to perform the methods provided in the various embodiments described above.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0164] The modules can be physically separate, for example, installed in different locations within a single device, installed on different devices, distributed across multiple network units, or distributed across multiple processors. Alternatively, the modules can be integrated, for example, installed in the same device, or integrated into a single codebase. The modules can exist in hardware form, software form, or a combination of both. This application can select some or all of the modules to achieve the objectives of this embodiment based on actual needs.

[0165] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An industrial anomaly detection method, characterized in that, The method includes: Acquire multimodal data, which includes two-dimensional image features and three-dimensional point cloud features; The two-dimensional image features and the three-dimensional point cloud features are input into a trained unsupervised feature fusion model, which outputs the target features; wherein, the unsupervised feature fusion model is used to extract and fuse the two-dimensional image features and the three-dimensional point cloud features. The target features are input into a preset decision model, and the results of the industrial product testing are obtained.

2. The method according to claim 1, characterized in that, The two-dimensional image features and the three-dimensional point cloud features are input into a trained unsupervised feature fusion model, which outputs target features, including: Align the three-dimensional point cloud features with the two-dimensional image features to obtain the two-dimensional point cloud features; The two-dimensional image features and the two-dimensional point cloud features are input into the trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs the target features.

3. The method according to claim 2, characterized in that, Aligning the three-dimensional point cloud features with the two-dimensional image features yields two-dimensional point cloud features, including: The three-dimensional point cloud features are processed using the farthest point sampling algorithm to obtain multiple center point features; Interpolation calculation is performed on the center point feature in the three-dimensional point cloud feature to obtain the three-dimensional center point feature corresponding to the center point feature; The three-dimensional center point features are projected onto a two-dimensional plane to obtain two-dimensional point cloud features.

4. The method according to claim 3, characterized in that, The 3D point cloud features are processed using the farthest point sampling algorithm to obtain multiple center point features, including: The three-dimensional point cloud features are sampled using the farthest point sampling algorithm to obtain N target point cloud features, where N is a positive integer; Based on the target point cloud features, the three-dimensional point cloud features are grouped to obtain N groups of point cloud features; Each set of point cloud features is encoded into a feature vector and input into a preset feature extraction model, so that the feature extraction model outputs the center point features corresponding to each set of point cloud features.

5. The method according to claim 3, characterized in that, Projecting the three-dimensional center point features onto a two-dimensional plane yields two-dimensional point cloud features, including: Using preset camera parameters, the three-dimensional center point features are projected onto a two-dimensional plane to obtain two-dimensional center point features corresponding to the three-dimensional center point features. If the two-dimensional center point feature matches a feature point in the two-dimensional image feature, then the two-dimensional center point feature is retained; If the two-dimensional center point feature does not match the feature point in the two-dimensional image feature, then the two-dimensional center point feature is set to 0; Average pooling is used to fill in the empty positions of the two-dimensional center point features to obtain two-dimensional point cloud features.

6. The method according to claim 1, characterized in that, The target features are input into a preset decision model, and the results are processed to obtain the detection results of industrial products, including: The target features are input into a preset decision model, which processes the target features to obtain an industrial product anomaly score and industrial product anomaly area information. The anomaly score is used to characterize the probability of an industrial product anomaly, and the industrial product anomaly area information is used to characterize the area where the probability of an industrial product anomaly is highest.

7. The method according to any one of claims 1-6, characterized in that, The multimodal data also includes sensor data; the method further includes: The sensor data is input into the preset decision model, which processes the sensor data to obtain abnormal operating status information of the industrial equipment.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: The unsupervised feature fusion model is trained using a patch-contrast loss function, and the learning rate is adjusted using a cosine function to obtain the trained unsupervised feature fusion model.

9. An industrial anomaly detection device, characterized in that, include: The acquisition module is used to acquire multimodal data, which includes two-dimensional image features and three-dimensional point cloud features; The processing module is used to input the two-dimensional image features and the three-dimensional point cloud features into a trained unsupervised feature fusion model, so that the unsupervised feature fusion model outputs target features; wherein, the unsupervised feature fusion model is used to extract and fuse features from the two-dimensional image features and the three-dimensional point cloud features; The processing module is also used to input the target features into a preset decision model and process them to obtain the detection results of industrial products.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to implement the industrial anomaly detection method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the industrial anomaly detection method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the industrial anomaly detection method according to any one of claims 1-8.

Images