Abnormality detection method and device and related equipment
By fusing features from RGB images, depth images, and point cloud data, and utilizing self-attention models and PointNet models for feature ranking and anomaly detection, this approach addresses the shortcomings of existing technologies in identifying complex anomalies, achieving efficient and interpretable industrial anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies are insufficient in identifying complex anomalies (such as dents, deformations, and spatial cracks) in industrial production, and are easily affected by factors such as material diversity, lighting changes, and occlusion, leading to detection errors.
By fusing features from RGB images, depth images, and point cloud data, and utilizing self-attention and PointNet models for feature ranking and anomaly detection, combined with Grad-CAM and SHAP frameworks for interpretable anomaly localization, we achieve collaborative enhancement of multimodal data and accurate inference of anomaly causes.
It improves the ability to identify complex anomalies, reduces detection errors, and further reduces errors by fusing multiple data types, thus achieving efficient identification of anomalies in industrial production processes and interpretable detection results.
Smart Images
Figure CN121767741A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to an anomaly detection method, apparatus and related equipment. Background Technology
[0002] In industrial production processes, electronic devices need to inspect for defects, quality issues, and equipment malfunctions. However, related technologies typically employ image-based vision methods for anomaly detection, such as using convolutional networks based on 2D images to detect scratches and flaws, or using depth cameras to acquire depth images for structural analysis. These methods often suffer from limited information, resulting in insufficient ability to identify complex anomalies (such as dents, deformations, and spatial cracks). Furthermore, these technologies are prone to detection errors due to the diverse materials of industrial components, variations in lighting, and occlusion. Summary of the Invention
[0003] This application provides an anomaly detection method, apparatus, and related equipment. Based on image anomaly data, feature sorting, and spatial anomaly data, the final anomaly detection result is obtained. The cause of the anomaly can be accurately inferred through multiple data, thereby improving the anomaly recognition capability and reducing the anomaly detection error.
[0004] In a first aspect, embodiments of this application provide an anomaly detection method, including: A first feature fusion vector is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. A second feature fusion vector is determined, which is obtained by adding noise to the first feature fusion vector; Based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image, abnormal image data is determined, and the abnormal image data corresponds to the at least one RGB image and the at least one depth image. Analyze the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature; Based on each point cloud data in the at least one point cloud data, determine the spatial anomaly data corresponding to each point cloud data; Based on the image anomaly data, the feature sorting, and the spatial anomaly data, anomaly detection results are obtained.
[0005] Optionally, obtaining the first feature fusion vector includes: Obtain the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; The first vector is input into the self-attention model to obtain the attention weights output by the self-attention model; the first vector is obtained by weighted summation of the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight. The first feature fusion vector is obtained based on the first vector and the attention weight.
[0006] Optionally, before obtaining the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight, the method further includes: Calculate the first mutual information between the first feature and the preset feature label, the second mutual information between the second feature and the preset feature label, and the third mutual information between the third feature and the preset feature label; The first weight is determined based on the first mutual information and the fourth mutual information; the fourth mutual information is the sum of the first mutual information, the second mutual information and the third mutual information. The second weight is determined based on the second mutual information and the fourth mutual information; The third weight is determined based on the third mutual information and the fourth mutual information.
[0007] Optionally, determining the second feature fusion vector includes: The mean and standard deviation are determined based on the first feature fusion vector; The target variable corresponding to the first feature fusion vector is determined by the mean and the standard deviation; the target variable is used to characterize the noise. The second feature fusion vector is obtained based on the target variable; The method further includes: Construct the loss function; If the loss value corresponding to the loss function is less than a preset loss threshold, the first feature fusion vector passes the verification. If the first feature fusion vector passes verification, the step of determining the abnormal image data corresponding to the at least one RGB image and the at least one depth image is executed based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image, and the second convolutional feature corresponding to the at least one depth image. The loss function is: ; in, The first feature fusion vector, This is the second feature fusion vector. For preset factor values, Let KL divergence be the KL divergence. The mean is... The standard deviation is... Characterization and The normal distribution formed It represents a standard normal distribution with a mean of 0 and a standard deviation of 1.
[0008] Optionally, determining the image anomalous data corresponding to the at least one RGB image and the at least one depth image based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, and the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image includes: The at least one RGB image is input into the DINO model to obtain the first convolutional feature output by the first preset convolutional layer in the DINO model; The at least one depth image is input into the 3DCNN model to obtain the second convolutional feature output by the second preset convolutional layer in the 3DCNN model; The abnormal region data in the image is determined using the first formula; A thermal image is determined using the first convolutional feature, the second convolutional feature, and the abnormal region data; The abnormal data in the image are determined using the thermal image; The first formula is: ; The preset number of features, The difference is... The sum of the first convolutional feature and the second convolutional feature. This refers to the abnormal region data.
[0009] Optionally, the step of analyzing the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature includes: The first feature fusion vector is input into the SHAP framework to obtain the predicted contribution values of the first feature, the second feature and the third feature output by the SHAP framework to the anomaly detection result; The feature ranking is determined based on the predicted contribution value.
[0010] Optionally, determining the spatial anomaly data corresponding to each point cloud data based on each point cloud data in the at least one point cloud data includes: The point cloud data is input into the PointNet model to obtain the local features of the corresponding points of the point cloud data output by the PointNet model. Based on the local features and anomaly weight vectors of the corresponding points in each point cloud data, the spatial anomaly data is determined.
[0011] Secondly, embodiments of this application provide an anomaly detection device, including: The acquisition module is used to acquire a first feature fusion vector, which is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. The first determining module is used to determine the second feature fusion vector, which is obtained by adding noise to the first feature fusion vector. The second determining module is used to determine abnormal image data based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image. The abnormal image data corresponds to the at least one RGB image and the at least one depth image. The analysis module is used to analyze the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature; The third determining module is used to determine the spatial anomaly data corresponding to each point cloud data based on each point cloud data in the at least one point cloud data. The fourth determining module is used to obtain anomaly detection results based on the image anomaly data, the feature sorting, and the spatial anomaly data.
[0012] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the anomaly detection method as described in the first aspect.
[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the anomaly detection method as described in the first aspect.
[0014] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the anomaly detection method as described in the first aspect.
[0015] In this embodiment, the electronic device acquires a first feature fusion vector, which is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. The electronic device then determines a second feature fusion vector by adding noise to the first feature fusion vector. Based on the difference between the second and first feature fusion vectors, a preset number of features, a first convolutional feature corresponding to at least one RGB image, and a second convolutional feature corresponding to at least one depth image, the electronic device determines image anomaly data. The electronic device then analyzes the first feature fusion vector to obtain a feature ranking among the first, second, and third features. Based on each point cloud data in at least one point cloud data set, the electronic device determines spatial anomaly data corresponding to each point cloud data set. Finally, the electronic device obtains the final anomaly detection result based on the image anomaly data, feature ranking, and spatial anomaly data. In this process, the electronic device can accurately infer the cause of the anomaly using multiple types of data (image anomaly data, feature ranking, and spatial anomaly data), improving the ability to identify complex anomalies, and reducing the error in anomaly detection by acquiring multiple types of data (image anomaly data, feature ranking, and spatial anomaly data). Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is one of the flowcharts of an anomaly detection method provided in the embodiments of this application; Figure 2 This is a flowchart of a data acquisition module provided in an embodiment of this application; Figure 3 This is a flowchart of a data preprocessing module provided in an embodiment of this application; Figure 4 This is a flowchart of a feature extraction module provided in an embodiment of this application; Figure 5 This is a flowchart of an adaptive feature fusion module provided in an embodiment of this application; Figure 6This is a second flowchart of an anomaly detection method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an anomaly detection device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] See Figure 1 , Figure 1 This is a flowchart of an anomaly detection method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step 101: Obtain the first feature fusion vector, which is obtained by fusing the first feature and first weight corresponding to at least one RGB image, the second feature and second weight corresponding to at least one depth image, and the third feature and third weight corresponding to at least one point cloud data.
[0020] In some embodiments, the electronic device can acquire at least one RGB image, at least one depth image, and at least one point cloud data through a data acquisition module. The depth image, also known as a range image, refers to an image where the distance (depth) from the image acquisition device (camera or radar) to various points in the scene is used as pixel values; it directly reflects the geometry of the visible surfaces of the scene. The point cloud data is a set of discrete points generated by a 3D scanning device, where each point contains at least 3D coordinate information, and some data also includes color or reflectance intensity attributes.
[0021] The data acquisition module's acquisition process can be as follows: Figure 2 As shown.
[0022] Step 201: Initialization and Equipment Calibration.
[0023] In some embodiments, the electronic device initializes and calibrates the data acquisition equipment, which may include: a high-resolution RGB industrial camera (with a resolution of at least 3648×5472 pixels), a TOF or structured light depth sensor, a laser scanner, and a turntable platform. The electronic device calibrates these devices, calculates intrinsic / extrinsic parameters (such as camera focal length and position) using a calibration board, and synchronizes clocks to ensure that the timestamps of all devices are consistent (error <1ms).
[0024] Step 202: Environmental preparation and object positioning.
[0025] In some embodiments, the electronic device can set ambient lighting (e.g., using controllable LEDs), place the industrial component to be inspected at a preset inspection position (e.g., the center of a turntable), and confirm that the position is stable and unobstructed. The electronic device then photographs the industrial component to be inspected, ensuring that RGB images, depth images, and point cloud data of the industrial component to be inspected can be acquired simultaneously.
[0026] Step 203: Acquire RGB image data.
[0027] In some embodiments, the electronic device triggers an RGB camera from a fixed viewing angle and acquires a sequence of color images. Each sample generates one RGB image in PNG or JPEG format, used to capture color, texture, and visible surface defects. After acquiring the images, the electronic device obtains an RGB image dataset.
[0028] Step 204: Acquire depth image data.
[0029] In some embodiments, the electronic device uses a depth sensor and a multi-light source system to capture depth images under broad illumination and outputs a TIFF format depth image. During this acquisition process, the electronic device can acquire surface details of the depth image (such as pits or deformations), bridge RGB textures and 3D geometry, and enhance anomaly localization accuracy.
[0030] Step 205: Collect point cloud data.
[0031] In some embodiments, the electronic device activates a laser scanner to perform a full scan of the components on the turntable, generating high-density point cloud data. The electronic device can also use structured light projection and triangulation to ensure coverage of all surfaces during the scanning process, outputting PCD format files that provide absolute geometric information, capturing anomalies such as overall deformation or spatial distribution, and the data supports 3D reconstruction.
[0032] Step 206: View synchronization and verification.
[0033] The electronic device ensures spatiotemporal alignment by aggregating all modal data (RGB image data, depth image data, and point cloud data). For example, the electronic device can use a synchronized view of the turntable angle to determine if alignment is correct, and performs preliminary verification. The electronic device can also check data integrity (e.g., missing rate <1%) and quality; if it fails to meet the requirements, the data is reacquired.
[0034] The electronic device can also preprocess the acquired RGB image data, depth image data, and point cloud data through a data preprocessing module. This module can standardize the processing of data from various modalities to reduce noise, eliminate differences between modalities, and enhance the efficiency of feature extraction. Specific techniques can be employed for different modalities (RGB images, depth images, point cloud data) to ensure that features between subsequent modalities can be fused.
[0035] The preprocessing process of the data preprocessing module can be as follows: Figure 3 As shown.
[0036] Step 301: Preprocess RGB image data.
[0037] The electronic device adjusts all RGB images to a uniform resolution (512×512 pixels) and adjusts the pixel value range to the [0, 1] interval to improve the stability and efficiency of the anomaly detection output process. The electronic device can apply mean-mode filtering to remove noise generated during imaging. Specifically, the electronic device can replace the value of each pixel with the median of its neighborhood (within a 3×3 window), thereby preserving the original color information while reducing noise, resulting in preprocessed RGB image data.
[0038] S302: Preprocess depth image data.
[0039] The electronic device first transforms the depth image to a unified coordinate system to ensure that the depth images are synchronized.
[0040] Optionally, electronic devices can unify the coordinate system using a coordinate transformation matrix, the formula of which can be: .in, This is the coordinate transformation matrix.
[0041] The formula for applying Gaussian filtering to denoise electronic devices can be: .in, For Gaussian filters, * indicates convolution operation.
[0042] Electronic devices further enhance surface features using photometric stereolithography. The formula can be: .in, It is a pixel intensity vector. This is the illumination direction matrix. The final result is the preprocessed depth image data.
[0043] S303: Preprocess point cloud data.
[0044] Electronic devices use statistical filters to select valid points.
[0045] Optionally, the electronic device uses a least squares algorithm to fit the planar model and removes outliers that deviate significantly from the model.
[0046] The electronic device then performs coordinate standardization on the point cloud data, using the following formula: .in, This represents the mean value of all points in the point cloud. This processing, where the standard deviation is used, ensures that all point cloud data are at the same scale.
[0047] The electronic device then performs density homogenization on the point cloud. This density homogenization can be achieved through voxel downsampling, resulting in preprocessed point cloud data.
[0048] After completing steps 301-303 above, the electronic device obtains the preprocessed RGB image, depth image and point cloud data. The electronic device can ensure that each modal data (RGB image, depth image and point cloud data) has good quality and consistency, providing reliable input for subsequent feature extraction and modality fusion.
[0049] Electronic devices can also extract useful features from pre-processed RGB images, depth images, and point cloud data through a feature extraction module to capture the shape, texture, and structural information of the industrial components to be inspected, in order to support subsequent anomaly detection.
[0050] The feature extraction process of the feature extraction module can be as follows: Figure 4 As shown.
[0051] Step 401: Extract the first feature of the RGB image.
[0052] Electronic devices can use the DINO model (Self-Distilling Unlabeled Learning Model) as the basis for feature extraction. For example, the DINO model can generate high-quality two-dimensional visual features through unsupervised learning methods, and the electronic device can obtain a 512-dimensional feature vector output by the DINO model. Its formula can be: .in, Indicates the extracted first Features of a frame RGB image Indicates the first Frame RGB image.
[0053] Step 402: Extract the second feature of the depth image.
[0054] For feature extraction from depth images, electronic devices can employ a 3D convolutional neural network (3DCNN) model. The 3DCNN model can process the spatial features of the depth map and extract surface geometric information.
[0055] Optionally, the electronic device inputs the preprocessed depth image into a 3DCNN model, which then performs multiple convolutions and pooling to output a 512-dimensional depth feature vector. The formula could be: .in, Indicates the first Feature extraction from frame depth images Indicates the first Frame depth images. In this way, electronic devices can effectively capture depth changes and structural features of an object's surface.
[0056] Step 403: Extract the third feature of the point cloud data.
[0057] Electronic devices employ the PointNet (point cloud network) model for feature extraction. The PointNet model is capable of processing unordered point cloud data, obtaining a high-dimensional representation by aggregating the local features of each point and extracting its global features.
[0058] Optionally, the global features of the point cloud can be obtained using the following formula: .in, Features of each point, For multilayer perceptron mapping, The number of points in the point cloud. For the first Frame point cloud data.
[0059] Through steps 401-403, the electronic device can obtain feature vectors for various modalities, ensuring that information from different modalities can be merged, making subsequent fusion processing more effective and ensuring that the advantages of each modality are fully integrated. Furthermore, the electronic device constructs dedicated feature extractors (such as DINO, 3DCNN, and PointNet) for each of the three modalities to extract complementary texture, geometric, and spatial structure features.
[0060] Electronic devices can also use an adaptive feature fusion module to fuse features from different modalities (including features from RGB images, depth images, and point clouds) into a unified feature representation, and achieve anomaly recognition and scoring based on deep learning methods.
[0061] In some embodiments, obtaining the first feature fusion vector includes: Obtain the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; The first vector is input into the self-attention model to obtain the attention weights output by the self-attention model; the first vector is obtained by weighted summation of the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight. The first feature fusion vector is obtained based on the first vector and the attention weight.
[0062] Understandably, in this embodiment, the electronic device forms a high-dimensional multimodal joint representation by deeply fusing the first feature, first weight, second feature, second weight, third feature, and third weight. While maintaining the independent expressive capabilities of each modality, it achieves information synergy enhancement, improving the system's ability to perceive and discriminate complex industrial anomalies.
[0063] The following is through Figure 5 The processing procedure of the adaptive feature fusion module is explained in detail.
[0064] Step 501: Calculate feature weights.
[0065] In some embodiments, before obtaining the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight, the method further includes: Calculate the first mutual information between the first feature and the preset feature label, the second mutual information between the second feature and the preset feature label, and the third mutual information between the third feature and the preset feature label; The first weight is determined based on the first mutual information and the fourth mutual information; the fourth mutual information is the sum of the first mutual information, the second mutual information and the third mutual information. The second weight is determined based on the second mutual information and the fourth mutual information; The third weight is determined based on the third mutual information and the fourth mutual information.
[0066] Optionally, for each modality's features, the electronic device calculates adaptive weights to determine the proportion of different modalities in the fusion process. In some embodiments, the electronic device can calculate the weights between different modalities by introducing mutual information, i.e., calculating the features of each modality. Preset feature labels corresponding to normal samples ( Mutual information between modalities is used to assess the importance of each modality. Indicates the first feature, Indicates the second characteristic, This indicates the third characteristic.
[0067] Its formula can be: ,in, These represent different modalities, where RGB images correspond to R, depth images to D, and point cloud data to P. The first, second, and third mutual information can be obtained using this formula.
[0068] The electronic device then defines the importance weights of the features using the fourth mutual information, and the formula can be: .in, This represents the first mutual information, the second mutual information, or the third mutual information, i.e. These represent the first mutual information, the second mutual information, or the third mutual information, respectively. This represents the fourth mutual information. This indicates the first weight, second weight, or third weight, that is, These represent the first weight of the RGB image, the second weight of the depth image, and the third weight of the point cloud data, respectively.
[0069] Understandably, in this embodiment, the electronic device quantifies the correlation between features of three modalities—RGB images, depth images, and point cloud data—and preset feature labels by introducing a feature importance evaluation method based on mutual information. The electronic device can calculate the contribution of each modality using mutual information scores and guide the weighted feature fusion process through normalized features, thereby achieving adaptive modality weight allocation. This embodiment effectively improves the task relevance of the fusion results, avoids modal redundancy or information conflicts, and is suitable for modal imbalance in multi-source perception tasks. Furthermore, this embodiment dynamically calculates modal weights based on the mutual information between modal features and the task objective, and can adaptively adjust modal contributions according to different scenarios and sample features, significantly enhancing the electronic device's ability to handle multi-source data differences, noise, and redundant features, and improving overall generalization performance and stability.
[0070] Step 502: Feature fusion.
[0071] Electronic devices combine the calculated weights with the feature vectors and achieve feature fusion through weighted summation or other nonlinear methods.
[0072] For example, electronic devices can calculate the first fused vector through weighted summation: ,in, Let be the first vector.
[0073] Electronic devices further enhance the model's expressive power by using a self-attention model to calculate the correlation between features. The calculation formula can be: .
[0074] The electronic device then combines the first vector with the attention weights to obtain the second vector: .
[0075] Step 503: Feature standardization.
[0076] The electronic device standardizes the fused second vector to ensure it has a uniform distribution and consistent scale, facilitating subsequent training and convergence. The formula can be: ;in, This is the first feature fusion vector.
[0077] Understandably, in steps 501-503, the electronic device jointly models the RGB image, depth image, and high-precision point cloud data. Through adaptive feature fusion and mutual information weighting mechanism, it achieves deep complementarity and synergistic enhancement of texture, structure, and spatial information, effectively improving the detection accuracy and robustness of complex anomalies (such as depressions, slight deformations, spatial structural anomalies, etc.).
[0078] Step 102: Determine the second feature fusion vector, which is obtained by adding noise to the first feature fusion vector.
[0079] The adaptive feature fusion module may also include the following processing.
[0080] In some embodiments, determining the second feature fusion vector includes: The mean and standard deviation are determined based on the first feature fusion vector; The target variable corresponding to the first feature fusion vector is determined by the mean and the standard deviation; the target variable is used to characterize the noise. The second feature fusion vector is obtained based on the target variable; The method further includes: Construct the loss function; If the loss value corresponding to the loss function is less than a preset loss threshold, the first feature fusion vector passes the verification. If the first feature fusion vector passes verification, the step of determining the abnormal image data corresponding to the at least one RGB image and the at least one depth image is executed based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image, and the second convolutional feature corresponding to the at least one depth image. The loss function is: ; in, The first feature fusion vector, This is the second feature fusion vector. For preset factor values, Let KL divergence be the KL divergence. The mean is... The standard deviation is... Characterization and The normal distribution formed It represents a standard normal distribution with a mean of 0 and a standard deviation of 1.
[0081] In this embodiment, the electronic device can perform anomaly detection on the first feature fusion vector.
[0082] Optionally, the electronic device can be designed with a variational autoencoder (VAE) structure, which consists of two main parts: an encoder and a decoder. The electronic device will... Input encoder, encoder will Compressing into the latent space yields the encoder's output latent variables (or target variables). mean and standard deviation .
[0083] Its formula can be: .
[0084] Electronic devices then use the following formula: Determine the target variable corresponding to the first feature fusion vector.
[0085] Electronic devices then use the following formula: From latent variables Reconstructing the second feature fusion vector .
[0086] The electronic device then combines the reconstruction loss (usually mean squared error) with the KL divergence of the latent space to obtain the loss function.
[0087] If the loss value corresponding to the loss function is less than the preset loss threshold, the first feature fusion vector passes verification, and the electronic device continues to execute the subsequent step 103. If the loss value is greater than or equal to the preset loss threshold, the first feature fusion vector fails verification, and the electronic device does not execute the subsequent step 103.
[0088] Understandably, in this embodiment, the electronic device learns the latent distribution of normal samples, performs auto-encoding and reconstruction of the input features, and then uses the reconstruction error to determine whether the input sample is abnormal. This embodiment completes the training of the entire industrial anomaly detection system.
[0089] In some embodiments, the electronic device can determine whether an industrial component under test has an anomaly by checking whether the difference between the second feature fusion vector and the first feature fusion vector is greater than a preset difference. If the difference is greater than the preset difference, an anomaly exists, and subsequent step 103 is executed; if the difference is less than or equal to the preset difference, no anomaly exists, and step 103 is not executed. That is, Electronic devices can be set with preset differences. ,like If the electronic device detects an abnormality, it will determine that the industrial component under test is abnormal; otherwise, it will be normal.
[0090] Step 103: Based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image, determine the abnormal image data, wherein the abnormal image data corresponds to the at least one RGB image and the at least one depth image.
[0091] Electronic devices can use an interpretable anomaly localization module to identify which features / regions in multimodal data caused the anomalies, and visualize this through heatmaps of the corresponding images. The main implementation paths include: image anomaly region localization (Grad-CAM), feature attribution analysis (SHAP), and point cloud anomaly region annotation.
[0092] In some embodiments, determining the image anomalous data corresponding to the at least one RGB image and the at least one depth image based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, and the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image includes: The at least one RGB image is input into the DINO model to obtain the first convolutional feature output by the first preset convolutional layer in the DINO model; The at least one depth image is input into the 3DCNN model to obtain the second convolutional feature output by the second preset convolutional layer in the 3DCNN model; The abnormal region data in the image is determined using the first formula; A thermal image is determined using the first convolutional feature, the second convolutional feature, and the abnormal region data; The abnormal data in the image are determined using the thermal image; The first formula is: ; The preset number of features, The difference is... The sum of the first convolutional feature and the second convolutional feature. This refers to the abnormal region data.
[0093] In this embodiment, the electronic device can utilize Grad-CAM (Gradient Weighted Class Activation Mapping) to locate anomalous regions in image modalities (RGB images and depth images). The electronic device can select the first convolutional feature in the last convolutional feature map of the DINO model, and the second convolutional feature in the last convolutional feature map of the 3DCNN model, and use the formula: The gradient of the feature map is backpropagated.
[0094] in, This can be the total number of elements in the last convolutional feature map of the DINO model and the last convolutional feature map of the 3DCNN model. This represents the gradient value.
[0095] Electronic devices can also be based on the formula: This yields a heatmap. This is a heat map.
[0096] Electronic devices will Interpolation is used to scale the image back to its original size and overlay it for display, enabling the salient localization of abnormal areas. In other words, electronic devices can identify image anomalies based on heatmaps.
[0097] Understandably, in this embodiment, the electronic device uses the difference between the second feature fusion vector and the first feature fusion vector, the number of features, and the first convolutional feature corresponding to at least one RGB image and the second convolutional feature corresponding to at least one depth image to determine image abnormal data from image modal data (RGB image and depth image), which facilitates the subsequent accurate acquisition of abnormal detection results.
[0098] Step 104: Analyze the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature.
[0099] Electronic devices can also determine the feature order among the first, second, and third features through an interpretable anomaly localization module.
[0100] In some embodiments, analyzing the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature includes: The first feature fusion vector is input into the SHAP framework to obtain the predicted contribution values of the first feature, the second feature and the third feature output by the SHAP framework to the anomaly detection result; The feature ranking is determined based on the predicted contribution value.
[0101] In this embodiment, the electronic device can use SHAP values from the SHAP (Shapley Additive Explanations) framework to attribute features to determine which fused feature dimensions or modal features have the greatest impact on anomaly detection. That is, the electronic device can use the SHAP framework to attribute the first feature fusion vector... Perform the analysis.
[0102] Optionally, for each dimension of input features (For example, the first feature, the second feature, or the third feature), the electronic device calculates its response to the anomaly detection result. The marginal contribution value is used to obtain the feature importance map (e.g., Top-k feature dimensions) output by the SHAP framework, which is sorted by absolute value. That is, the feature ranking is obtained.
[0103] Understandably, in this embodiment, the electronic device can obtain accurate anomaly detection results by acquiring feature sorting.
[0104] Step 105: Based on each point cloud data in the at least one point cloud data, determine the spatial anomaly data corresponding to each point cloud data.
[0105] Electronic devices can also identify spatial anomaly data through an interpretable anomaly localization module.
[0106] In some embodiments, determining the spatial anomaly data corresponding to each point cloud data based on each point cloud data in the at least one point cloud data includes: The point cloud data is input into the PointNet model to obtain the local features of the corresponding points of the point cloud data output by the PointNet model. Based on the local features and anomaly weight vectors of the corresponding points in each point cloud data, the spatial anomaly data is determined.
[0107] Electronic devices can retain the activation values of each point in the point cloud data from the global features output by the PointNet model, and process each point... The formula for calculating the abnormal contribution score can be: .
[0108] in, For local features of a point, The weight vector for the abnormal direction (or the abnormal weight vector). An anomaly contribution score is assigned to each point. Electronic devices can identify spatial anomalies based on the anomaly contribution scores for all points.
[0109] Understandably, in this embodiment, the electronic device can obtain spatial anomaly data to facilitate accurate subsequent anomaly detection results.
[0110] Step 106: Based on the image anomaly data, the feature sorting, and the spatial anomaly data, obtain the anomaly detection result.
[0111] In some embodiments, the electronic device can also achieve cross-modal interpretation alignment through an interpretable anomaly localization module. That is, the interpretable anomaly localization module on the electronic device can project image anomaly data, feature ranking, and spatial anomaly data into a shared spatial coordinate system, such as the image plane or 3D world coordinates, to achieve unified alignment and visualization of multimodal interpretation results in space, and obtain the anomaly detection results output by the interpretable anomaly localization module.
[0112] The above embodiments construct a causal visualization alignment mechanism that integrates SHAP (Feature Attribution), Grad-CAM (Image Saliency Heatmap), and point cloud heatmap. This mechanism can track key features and modal sources for anomaly detection and visualize high-impact factor regions in image or point cloud space, achieving multi-level causal explanations based on anomaly localization, feature dimensions, and spatial location. These embodiments significantly improve the interpretability of electronic devices, supporting industrial engineers in conducting causal analysis and decision verification for model judgments. Furthermore, by combining Grad-CAM, SHAP, and spatial projection technologies, these embodiments map salient regions of RGB images, depth images, and point cloud data to a unified space, outputting interpretable anomaly detection results. This achieves visualized, spatially aligned, and semantically consistent reliable explanation results, meeting the high controllability and traceability requirements in industrial applications.
[0113] As described in steps 101-106 above, the electronic device acquires a first feature fusion vector, which is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. The electronic device then determines a second feature fusion vector by adding noise to the first feature fusion vector. Based on the difference between the second and first feature fusion vectors, a preset number of features, a first convolutional feature corresponding to at least one RGB image, and a second convolutional feature corresponding to at least one depth image, the electronic device identifies image anomaly data. The electronic device then analyzes the first feature fusion vector to obtain a feature ranking among the first, second, and third features. Based on each point cloud data in at least one point cloud data set, the electronic device identifies spatial anomaly data corresponding to each point cloud data set. Finally, the electronic device obtains the final anomaly detection result based on the image anomaly data, the feature ranking, and the spatial anomaly data. In this process, electronic devices can accurately infer the cause of anomalies through various types of data (image anomaly data, feature ranking, and spatial anomaly data), improve the ability to identify complex anomalies, and reduce the error of anomaly detection by acquiring various types of data (image anomaly data, feature ranking, and spatial anomaly data).
[0114] In some embodiments, the image modal data in this application, such as RGB images and depth images, can be replaced with other image modal data, such as infrared images, thermal imaging images, etc., without limitation.
[0115] In some embodiments, electronic devices may also perform multimodal fusion using a Transformer (self-attention mechanism) model, without limitation.
[0116] In some embodiments, electronic devices may also use MAE (mask autoencoder) models, Diffusion models (or diffusion models) to determine the second feature fusion vector, without any limitation.
[0117] The following is through Figure 6 The process of embodiments of this application will be described.
[0118] Step 601: Identify the industrial component to be tested.
[0119] Step 602: Determine at least one RGB image, at least one depth image, and at least one point cloud data of the industrial component to be inspected.
[0120] Step 603: Preprocess at least one RGB image, at least one depth image, and at least one point cloud data.
[0121] Step 604: Determine the first feature corresponding to at least one RGB image using the DINO model.
[0122] Step 605: Determine the second feature corresponding to at least one depth image using a 3DCNN model.
[0123] Step 606: Determine at least one third feature corresponding to the point cloud data using the PointNet model.
[0124] Step 607: Determine the first weight corresponding to at least one RGB image, the second weight corresponding to at least one depth image, and the third weight corresponding to at least one point cloud data.
[0125] Step 608: Obtain the first feature fusion vector based on the first feature, first weight, second feature, second weight, third feature, and third weight.
[0126] Step 609: Obtain the second feature fusion vector through the VAE structure, and determine the difference between the second feature fusion vector and the first feature fusion vector.
[0127] If the difference is greater than the preset difference, there is an anomaly, and the subsequent step 610 is executed. If the difference is less than or equal to the preset difference, there is no anomaly, and step 610 is not executed.
[0128] Step 610: Determine the image anomaly data, feature ranking, and spatial anomaly data, and obtain the anomaly detection results based on the image anomaly data, feature ranking, and spatial anomaly data.
[0129] Please refer to Figure 7 , Figure 7 This is a schematic diagram of an anomaly detection device 700 according to an embodiment of this application. The anomaly detection device 700 includes: The acquisition module 701 is used to acquire a first feature fusion vector, which is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. The first determining module 702 is used to determine the second feature fusion vector, which is obtained by adding noise to the first feature fusion vector. The second determining module 703 is used to determine abnormal image data based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image. The abnormal image data corresponds to the at least one RGB image and the at least one depth image. Analysis module 704 is used to analyze the first feature fusion vector to obtain the feature ranking among the first feature, the second feature and the third feature; The third determining module 705 is used to determine the spatial anomaly data corresponding to each point cloud data based on each point cloud data in the at least one point cloud data. The fourth determining module 706 is used to obtain anomaly detection results based on the image anomaly data, the feature sorting, and the spatial anomaly data.
[0130] Optionally, the acquisition module 701 may also include: The first acquisition unit is used to acquire the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; The first determining unit is used to input the first vector into the self-attention model to obtain the attention weights output by the self-attention model; the first vector is obtained by weighted summation of the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; The second determining unit is used to obtain the first feature fusion vector based on the first vector and the attention weight.
[0131] Optionally, the anomaly detection device 700 may also include: The calculation module is used to calculate the first mutual information between the first feature and the preset feature label, the second mutual information between the second feature and the preset feature label, and the third mutual information between the third feature and the preset feature label; The fifth determining module is used to determine the first weight based on the first mutual information and the fourth mutual information; the fourth mutual information is the sum of the first mutual information, the second mutual information and the third mutual information; The sixth determining module is used to determine the second weight based on the second mutual information and the fourth mutual information; The seventh determining module is used to determine the third weight based on the third mutual information and the fourth mutual information.
[0132] Optionally, the first determining module 702 may further include: The third determining unit is used to determine the mean and standard deviation based on the first feature fusion vector; The fourth determining unit is used to determine the target variable corresponding to the first feature fusion vector through the mean and the standard deviation; the target variable is used to characterize the noise; The fifth determining unit is used to obtain the second feature fusion vector based on the target variable; The anomaly detection device 700 may also include: Modules for building loss functions; The verification module is used to verify the first feature fusion vector when the loss value corresponding to the loss function is less than a preset loss threshold. An execution module is configured to, when the first feature fusion vector passes verification, execute the step of determining abnormal image data corresponding to the at least one RGB image and the at least one depth image based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, the first convolutional feature corresponding to the at least one RGB image, and the second convolutional feature corresponding to the at least one depth image. The loss function is: ; in, The first feature fusion vector, This is the second feature fusion vector. For preset factor values, Let KL divergence be the KL divergence. The mean is... The standard deviation is... Characterization and The normal distribution formed It represents a standard normal distribution with a mean of 0 and a standard deviation of 1.
[0133] Optionally, the second determining module 703 may further include: The sixth determining unit is used to input the at least one RGB image into the DINO model to obtain the first convolutional feature output by the first preset convolutional layer in the DINO model; The seventh determining unit is used to input the at least one depth image into the 3DCNN model to obtain the second convolutional feature output by the second preset convolutional layer in the 3DCNN model; The eighth determining unit is used to determine abnormal region data in the image using the first formula; The ninth determining unit is used to determine the thermal image using the first convolutional feature, the second convolutional feature, and the abnormal region data; The tenth determining unit is used to determine the image abnormal data through the thermal image; The first formula is: ; The preset number of features, The difference is... The sum of the first convolutional feature and the second convolutional feature. This refers to the abnormal region data.
[0134] Optionally, the analysis module 704 may also include: The eleventh determining unit is used to input the first feature fusion vector into the SHAP framework to obtain the predicted contribution values of the first feature, the second feature and the third feature output by the SHAP framework to the anomaly detection result; The twelfth determining unit is used to determine the feature ranking based on the predicted contribution value.
[0135] Optionally, the third determining module 705 may also include: The thirteenth determining unit is used to input the point cloud data into the PointNet model to obtain the local features of the corresponding points of the point cloud data output by the PointNet model. The fourteenth determining unit is used to determine the spatial abnormal data based on the local features and abnormal weight vectors of the corresponding points of each point cloud data.
[0136] The anomaly detection device 700 provided in this application embodiment can perform the above-described... Figure 1 The method embodiments shown are similar in principle and technical effect, and will not be described again here.
[0137] This application also provides an electronic device. Since the principle by which this electronic device solves the problem is similar to the anomaly detection method in this application, the implementation of this electronic device can be found elsewhere. Figure 1 The implementation of the method shown will not be repeated here. Figure 8 As shown, the electronic device according to an embodiment of this application includes: a processor 810, configured to read a program from a memory 820 and execute the following processes: A first feature fusion vector is obtained by fusing a first feature and a first weight corresponding to at least one RGB image, a second feature and a second weight corresponding to at least one depth image, and a third feature and a third weight corresponding to at least one point cloud data. A second feature fusion vector is determined, which is obtained by adding noise to the first feature fusion vector; Based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image, abnormal image data is determined, and the abnormal image data corresponds to the at least one RGB image and the at least one depth image. Analyze the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature; Based on each point cloud data in the at least one point cloud data, determine the spatial anomaly data corresponding to each point cloud data; Based on the image anomaly data, the feature sorting, and the spatial anomaly data, anomaly detection results are obtained.
[0138] Optionally, the processor 810 is also used to read the program in the memory 820 and perform the following steps: obtaining the first feature fusion vector includes: Obtain the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; The first vector is input into the self-attention model to obtain the attention weights output by the self-attention model; the first vector is obtained by weighted summation of the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight. The first feature fusion vector is obtained based on the first vector and the attention weight.
[0139] Optionally, the processor 810 is also configured to read the program from the memory 820 and further perform the following steps: before acquiring the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight, the method further includes: Calculate the first mutual information between the first feature and the preset feature label, the second mutual information between the second feature and the preset feature label, and the third mutual information between the third feature and the preset feature label; The first weight is determined based on the first mutual information and the fourth mutual information; the fourth mutual information is the sum of the first mutual information, the second mutual information and the third mutual information. The second weight is determined based on the second mutual information and the fourth mutual information; The third weight is determined based on the third mutual information and the fourth mutual information.
[0140] Optionally, the processor 810 is also used to read the program in the memory 820 and perform the following steps: determining the second feature fusion vector includes: The mean and standard deviation are determined based on the first feature fusion vector; The target variable corresponding to the first feature fusion vector is determined by the mean and the standard deviation; the target variable is used to characterize the noise. The second feature fusion vector is obtained based on the target variable; The processor 810 is also used to read the program from the memory 820 and to perform the following steps: constructing a loss function; If the loss value corresponding to the loss function is less than a preset loss threshold, the first feature fusion vector passes the verification. If the first feature fusion vector passes verification, the step of determining the abnormal image data corresponding to the at least one RGB image and the at least one depth image is executed based on the difference between the second feature fusion vector and the first feature fusion vector, the preset number of features, the first convolutional feature corresponding to the at least one RGB image, and the second convolutional feature corresponding to the at least one depth image. The loss function is: ; in, The first feature fusion vector, This is the second feature fusion vector. For preset factor values, Let KL divergence be the KL divergence. The mean is... The standard deviation is... Characterization and The normal distribution formed It represents a standard normal distribution with a mean of 0 and a standard deviation of 1.
[0141] Optionally, the processor 810 is further configured to read the program in the memory 820 and perform the following steps: determining the image aberration data corresponding to the at least one RGB image and the at least one depth image based on the difference between the second feature fusion vector and the first feature fusion vector, a preset number of features, and the first convolutional feature corresponding to the at least one RGB image and the second convolutional feature corresponding to the at least one depth image, including: The at least one RGB image is input into the DINO model to obtain the first convolutional feature output by the first preset convolutional layer in the DINO model; The at least one depth image is input into the 3DCNN model to obtain the second convolutional feature output by the second preset convolutional layer in the 3DCNN model; The abnormal region data in the image is determined using the first formula; A thermal image is determined using the first convolutional feature, the second convolutional feature, and the abnormal region data; The abnormal data in the image are determined using the thermal image; The first formula is: ; The preset number of features, The difference is... The sum of the first convolutional feature and the second convolutional feature. This refers to the abnormal region data.
[0142] Optionally, the processor 810 is also configured to read the program in the memory 820 and perform the following steps: analyzing the first feature fusion vector to obtain the feature ranking among the first feature, the second feature, and the third feature, including: The first feature fusion vector is input into the SHAP framework to obtain the predicted contribution values of the first feature, the second feature and the third feature output by the SHAP framework to the anomaly detection result; The feature ranking is determined based on the predicted contribution value.
[0143] Optionally, the processor 810 is further configured to read the program in the memory 820 and perform the following steps: determining the spatial anomaly data corresponding to each point cloud data based on each point cloud data in the at least one point cloud data, including: The point cloud data is input into the PointNet model to obtain the local features of the corresponding points of the point cloud data output by the PointNet model. Based on the local features and anomaly weight vectors of the corresponding points in each point cloud data, the spatial anomaly data is determined.
[0144] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 810 and memory represented by memory 820 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface.
[0145] The electronic device provided in this application embodiment can perform the above-described functions. Figure 1 The method embodiments shown are similar in principle and technical effect, and will not be described again here.
[0146] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described anomaly detection method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0147] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described anomaly detection method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0150] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. An anomaly detection method characterized by, The method comprises: obtaining a first feature fusion vector, the first feature fusion vector being obtained by fusing a first feature corresponding to at least one RGB image and a first weight, a second feature corresponding to at least one depth image and a second weight, and a third feature corresponding to at least one point cloud data and a third weight; determining a second feature fusion vector, the second feature fusion vector being obtained by adding noise to the first feature fusion vector; determining image abnormal data based on a difference between the second feature fusion vector and the first feature fusion vector, a preset feature quantity, a first convolutional feature corresponding to the at least one RGB image, and a second convolutional feature corresponding to the at least one depth image, the image abnormal data corresponding to the at least one RGB image and the at least one depth image; analyzing the first feature fusion vector to obtain a feature ranking between the first feature, the second feature, and the third feature; determining spatial abnormal data corresponding to each of the at least one point cloud data based on the each of the at least one point cloud data; obtaining an anomaly detection result based on the image abnormal data, the feature ranking, and the spatial abnormal data.
2. The method of claim 1, wherein, The method comprises: obtaining the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; inputting a first vector into a self-attention model to obtain an attention weight output by the self-attention model, the first vector being obtained by weighted summation of the first feature, the first weight, the second feature, the second weight, the third feature, and the third weight; obtaining the first feature fusion vector based on the first vector and the attention weight.
3. The method of claim 2, wherein, Before the method comprises: calculating a first mutual information between the first feature and a preset feature label, a second mutual information between the second feature and the preset feature label, and a third mutual information between the third feature and the preset feature label; determining the first weight based on the first mutual information and a fourth mutual information, the fourth mutual information being a sum of the first mutual information, the second mutual information, and the third mutual information; determining the second weight based on the second mutual information and the fourth mutual information; determining the third weight based on the third mutual information and the fourth mutual information.
4. The method of claim 1, wherein, The method comprises: determining a mean value and a standard deviation based on the first feature fusion vector; determining a target variable corresponding to the first feature fusion vector based on the mean value and the standard deviation, the target variable being used to represent the noise; obtaining the second feature fusion vector based on the target variable; The method further comprises: constructing a loss function; in a case where a loss value corresponding to the loss function is less than a preset loss threshold, the first feature fusion vector passes the verification; In a case where the first feature fusion vector passes verification, performing the step of determining image abnormal data corresponding to the at least one RGB image and the at least one depth image based on a difference between the second feature fusion vector and the first feature fusion vector, a preset feature quantity, first convolution features corresponding to the at least one RGB image, and second convolution features corresponding to the at least one depth image; The loss function is: ; wherein, is the first feature fusion vector, is the second feature fusion vector, is a preset factor value, is a KL divergence, is the mean, is the standard deviation, characterizes and a normal distribution, characterizes a standard normal distribution with a mean of 0 and a standard deviation of 1.
5. The method of claim 1, wherein, The step of determining image abnormal data corresponding to the at least one RGB image and the at least one depth image based on a difference between the second feature fusion vector and the first feature fusion vector, a preset feature quantity, first convolution features corresponding to the at least one RGB image, and second convolution features corresponding to the at least one depth image includes: inputting the at least one RGB image into a DINO model to obtain the first convolution features output by a first preset convolution layer in the DINO model; inputting the at least one depth image into a 3DCNN model to obtain the second convolution features output by a second preset convolution layer in the 3DCNN model; determining abnormal region data in the image through a first formula; determining a heat image based on the first convolution features, the second convolution features, and the abnormal region data; determining the image abnormal data based on the heat image; wherein the first formula is: ; is the preset feature quantity, is the difference value, is the sum of the first convolutional feature and the second convolutional feature, is the abnormal region data.
6. The method of claim 1, wherein, The step of analyzing the first feature fusion vector to obtain a feature ranking among the first feature, the second feature, and the third feature includes: inputting the first feature fusion vector into a SHAP framework to obtain a prediction contribution value of the first feature, the second feature, and the third feature to the abnormal detection result output by the SHAP framework; determining the feature ranking based on the prediction contribution value.
7. The method of claim 1, wherein, The step of determining spatial abnormal data corresponding to each point cloud data in the at least one point cloud data includes: inputting the each point cloud data into a PointNet model to obtain local features of points corresponding to the each point cloud data output by the PointNet model; determining the spatial abnormal data based on the local features of points corresponding to the each point cloud data and an abnormal weight vector.
8. An abnormality detection device characterized by comprising: The method includes: an acquisition module configured to acquire a first feature fusion vector, the first feature fusion vector being obtained by fusing a first feature corresponding to at least one RGB image and a first weight, a second feature corresponding to at least one depth image and a second weight, and a third feature corresponding to at least one point cloud data and a third weight; a first determination module configured to determine a second feature fusion vector, the second feature fusion vector being obtained by adding noise to the first feature fusion vector; a second determination module configured to determine image abnormal data based on a difference between the second feature fusion vector and the first feature fusion vector, a preset feature quantity, first convolution features corresponding to the at least one RGB image, and second convolution features corresponding to the at least one depth image, the image abnormal data corresponding to the at least one RGB image and the at least one depth image; an analysis module, configured to analyze the first feature fusion vector to obtain a feature ranking between the first feature, the second feature and the third feature; a third determination module, configured to determine, based on each point cloud data in the at least one point cloud data, spatial anomaly data corresponding to the each point cloud data; a fourth determination module, configured to obtain an anomaly detection result based on the image anomaly data, the feature ranking and the spatial anomaly data.
9. An electronic device, comprising: comprising: a processor, a memory, and a program stored on the memory and capable of running on the processor, the program being executed by the processor to implement the steps in the anomaly detection method according to any one of claims 1 to 7.
10. A computer readable storage medium for storing a computer program, characterized in that, the computer program being executed by the processor to implement the steps in the anomaly detection method according to any one of claims 1 to 7.
11. A computer program product, characterised in that, comprising computer instructions, the computer instructions being executed by the processor to implement the steps in the anomaly detection method according to any one of claims 1 to 7.