An industrial internet of things lightweight anomaly detection method based on abnormal simulation and hierarchical multi-modal feature fusion and a detection system thereof
By employing generative anomaly simulation and hierarchical multimodal feature fusion, the problems of detection accuracy and computational efficiency in the Industrial Internet of Things (IIoT) are solved, achieving efficient detection of minute defects, which is suitable for resource-constrained equipment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEILONGJIANG UNIV
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-16
AI Technical Summary
Existing industrial IoT anomaly detection methods are not accurate enough when detecting minute geometric defects, have high computational overhead for multimodal fusion, high inference latency, and are difficult to deploy on resource-constrained devices.
A generative anomaly enhancement strategy is adopted to synthesize pseudo-anomaly samples. A lightweight dual-branch reconstruction network and a hierarchical cross-modal attention fusion module are used to reconstruct and fuse features of RGB images and depth maps respectively, generating fused feature maps. Pixel-level anomaly classification is then performed through a segmentation network.
It improves detection performance by 1.7%, fills the detection blind spot of single mode, increases detection accuracy by 7.5%, and achieves an inference speed of 20.91 FPS, making it suitable for resource-constrained edge devices.
Smart Images
Figure CN122222984A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a lightweight anomaly detection method and system for industrial Internet of Things based on anomaly simulation and hierarchical multimodal feature fusion, belonging to the field of industrial automation detection technology. Background Technology
[0002] The Industrial Internet of Things (IIoT) connects sensors, instruments, and equipment to the internet to enable real-time monitoring and data-driven analysis of industrial production processes, and has become one of the core technologies for intelligent manufacturing and industrial automation. In product quality inspection, traditional manual inspection methods are gradually being replaced by intelligent vision inspection systems, especially in high-end manufacturing fields such as precision electronics and automotive parts. The presence of defects can severely impact product functionality and lifespan, thus placing extremely high demands on inspection accuracy and efficiency.
[0003] However, anomalous samples in industrial settings are typically extremely rare and diverse in form, making it difficult to obtain large-scale labeled data. Therefore, unsupervised anomaly detection has become a research hotspot. Most existing unsupervised methods are based on RGB images for modeling, such as feature embedding methods, memory-based methods, and reconstruction-based methods. While these methods have made progress to some extent, RGB images often fail to provide sufficient discriminative information when dealing with subtle, localized, or geometrically distinctive defects commonly found in industrial environments.
[0004] In recent years, with the release of 3D datasets such as MVTec 3D-AD, researchers have begun to explore multimodal anomaly detection methods that combine RGB images with 3D point clouds. These methods significantly improve detection robustness by fusing visual and geometric information. However, existing multimodal methods generally rely on large-scale pre-trained backbone networks (such as ResNet and SwingTransformer) and complex memory mechanisms, resulting in large model parameter counts, high inference latency, and severe computational resource consumption, making them difficult to deploy in resource-constrained IIoT edge devices. Furthermore, the fundamental differences in data format, resolution, and spatial structure between RGB images and 3D point clouds lead to alignment difficulties and information loss during cross-modal feature fusion, further limiting the model's detection performance.
[0005] In summary, existing technologies still have shortcomings in terms of detection accuracy, computational efficiency, and modal fusion. There is an urgent need for a lightweight, efficient, and suitable multimodal unsupervised anomaly detection method for industrial IoT environments. Summary of the Invention
[0006] The purpose of this invention is to address the problems of existing industrial IoT anomaly detection methods, such as insufficient ability to detect subtle geometric defects, high computational overhead of multimodal fusion, high inference latency, and difficulty in deployment on resource-constrained devices. This invention provides a lightweight industrial IoT anomaly detection method and system based on anomaly simulation and hierarchical multimodal feature fusion.
[0007] The present invention discloses a lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion, which includes the following steps:
[0008] Step 1: Use the RGB image and depth map of normal products collected by industrial IoT sensors as normal samples; adopt a generative anomaly enhancement strategy to synthesize pseudo-anomaly samples in the foreground region of normal samples, and construct a multimodal training set containing normal samples and pseudo-anomaly samples.
[0009] Step 2: Train a lightweight dual-branch reconstruction network using a multimodal training set to perform defect-free reconstruction of the RGB image and depth map respectively, and obtain the reconstructed image corresponding to the input image.
[0010] Step 3: Extract multi-level features from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map respectively, and fuse the features of each modality through the hierarchical cross-modal attention fusion module to generate a fused feature map;
[0011] Step 4: Input the fused feature map into the segmentation network, perform pixel-level anomaly classification, and output the segmentation results of the anomaly region.
[0012] Preferably, the specific method for synthesizing pseudo-abnormal samples in the foreground region of normal samples using the generative anomaly enhancement strategy described in step 1 includes:
[0013] Generate anomaly location masks using Perlin noise. ;
[0014] Extracting grayscale texture images from natural images ;
[0015] set up To be from the interval Uniformly sampled random numbers, The preset anomaly simulation probability;
[0016] For depth map First extract the foreground mask. The abnormal location mask is combined with the foreground mask to obtain the foreground abnormality mask. ;
[0017] if Then, anomaly simulation is performed to generate a pseudo-anomaly depth map. :
[0018] ;
[0019] in, To be from the interval Opacity parameter for uniform sampling Represents the Hadamard product;
[0020] when hour, ;
[0021] For RGB images Using the same foreground mask Generate a pseudo-abnormal RGB image.
[0022] Preferably, the lightweight dual-branch reconstruction network in step 2 adopts the Swin-Unet network with Swin Transformer as the backbone to reconstruct the RGB image and the depth map respectively.
[0023] A spatially weighted loss function is introduced during the reconstruction process to enhance the modeling ability of foreground defect regions;
[0024] Total reconstruction losses obtained Represented as:
[0025] ;
[0026] in, Indicates the input image. Indicates the reconstructed image. This represents the loss balance hyperparameter. This represents the total number of pixels in the image. This represents the cumulative sum of pixel-level reconstruction losses in the background region. This represents the cumulative sum of pixel-level reconstruction losses in the foreground region. Represents structural similarity loss;
[0027] in:
[0028] ;
[0029] ;
[0030] This represents the set of pixels in an image that belong to the background region. This represents the set of pixels in the image that belong to the foreground region. Indicates the position index of the pixel. Indicates the pixel position of the input image. Pixel value at that location, Indicates the pixel location of the reconstructed image. Pixel value at that location, Indicates the pixel position The L2 loss between the input image and the reconstructed image is calculated.
[0031] Preferably, the lightweight dual-branch reconstruction network described in step 2 is trained with the discriminator generative adversarial network;
[0032] The discriminator uses a PatchGAN structure to distinguish between real samples and reconstructed samples.
[0033] The overall loss function of a generative adversarial network is expressed as a weighted sum of the adversarial losses of the RGB branch and the depth branch:
[0034] ;
[0035] in, This represents the adversarial loss function of the generator. This represents the loss balance coefficient for the RGB branch. This represents the loss balance coefficient for deep branches. Represents the mathematical expectation. This indicates the discriminator output for the RGB branch. The discriminator output represents the depth branch. The generator output represents the RGB branch. The generator output represents the depth branch. Indicates the input sample;
[0036] ;
[0037] in, This represents the total loss function of the generator. Indicates the losses incurred during reconstruction;
[0038]
[0039] in, This represents the adversarial loss function of the discriminator. Represents the actual samples of the RGB branch. This represents a real sample representing a deep branch.
[0040] Preferably, the , .
[0041] Preferably, the hierarchical cross-modal attention fusion module in step 3 includes: a single-modal feature extraction unit, an intra-modal feature enhancement unit, a hierarchical fusion unit, and a cross-modal attention unit;
[0042] The single-modal feature extraction unit uses a pre-trained ResNet50 as the backbone network to extract the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, respectively. Figure 4 The output of the first four residual blocks of each input is used to obtain multi-level semantic features;
[0043] The intramodal feature enhancement unit employs a dual attention module (DAM) obtained by combining the channel attention module (CAM) and the spatial attention module (SAM) to enhance the features at each level of each modality. The enhanced features are as follows:
[0044] ;
[0045] in, This represents the feature map of any mode at a certain level. This represents the Dual Attention Module (DAM). This represents the feature map after enhancement by the dual attention module;
[0046] The hierarchical fusion unit first performs element-wise addition and channel-wise concatenation fusion on the original RGB and reconstructed RGB feature maps to obtain an RGB fused feature map; simultaneously, it performs the same fusion operation on the original depth and reconstructed depth feature maps to obtain a depth fused feature map.
[0047] ;
[0048] ;
[0049] ;
[0050] ;
[0051] in, This represents the feature map of the original RGB image after enhancement by the dual attention module. This represents the feature map of the reconstructed RGB image after enhancement by the dual attention module. This represents the intermediate fused feature map obtained by adding RGB modes. This represents the intermediate fused feature map obtained by concatenating RGB modes. This represents the feature map after the original depth map has been enhanced by the dual attention module. This represents the feature map after the reconstructed depth map has been enhanced by the dual attention module. This represents the intermediate fused feature map obtained by adding deep modes. This represents the intermediate fused feature map obtained by concatenating deep modes;
[0052] The cross-modal attention unit projects the RGB fused feature map and the deep fused feature map through convolutional layers, calculates their correlation matrix, normalizes it using softmax to obtain attention weights, and then performs a weighted sum of the RGB fused feature map and the deep fused feature map according to the weights to generate the final fused feature map.
[0053] ;
[0054] in, This represents the final output fused feature map. This means adding the weighted RGB features to the depth features element by element. This represents the cross-modal attention function.
[0055] Preferably, the channel attention module performs average pooling and max pooling on the feature map along the channel dimension, concatenates the results, and then generates spatial attention weights through a convolutional layer.
[0056] Preferably, the segmentation network in step 4 adopts a U-Net structure, increasing the number of skip connections in the encoder, introducing transposed convolutions and upsampling in the decoder, and adding dilated convolutional layers to expand the receptive field; the segmentation network is trained using a focal loss function, which is expressed as:
[0057] ;
[0058] in, This represents the actual label mask. This represents the output predicted probability map of the segmentation network. This represents the predicted probability that each pixel belongs to the true class. Represents the category balance coefficient. This indicates the focus parameter.
[0059] Preferably, the lightweight dual-branch reconstruction network and the segmentation network employ a joint optimization strategy, with optimization objectives including reconstruction loss and classification loss, and the total loss function being:
[0060] ;
[0061] in, Represents the total loss function. This represents the reconstruction loss of the RGB branch. This represents the reconstruction loss for deep branches. Indicates focal loss. This represents the loss balance hyperparameter.
[0062] The present invention proposes a lightweight anomaly detection system for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion, which includes:
[0063] The data augmentation module is used to perform the anomaly simulation step. It takes the RGB image and depth map of normal products collected by industrial IoT sensors as input, and uses a generative anomaly augmentation strategy to synthesize pseudo-anomaly samples in the foreground region of normal samples to build a multimodal training set containing normal samples and pseudo-anomaly samples.
[0064] The image reconstruction module is used to perform image reconstruction steps. It uses a lightweight dual-branch reconstruction network to perform defect-free reconstruction of the RGB image and the depth map respectively, and obtains the reconstructed image corresponding to the input image.
[0065] The feature fusion module is used to perform the full feature fusion step, extracting multi-level features from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, and then fusing the features of each modality through the hierarchical cross-modal attention fusion module to generate a fused feature map;
[0066] The anomaly segmentation module is used to perform the anomaly localization step. It inputs the fused feature map into the segmentation network, performs pixel-level anomaly classification, and outputs the segmentation results of the anomaly region.
[0067] Advantages of this invention: The lightweight anomaly detection method and system for industrial IoT proposed in this invention, based on anomaly simulation and hierarchical multimodal feature fusion, synthesizes diverse pseudo-anomalies on normal samples by fusing Perlin noise and natural textures, effectively solving the problem of scarce anomaly samples in industrial scenarios and improving detection performance by 1.7%. Simultaneously, by utilizing RGB images and depth maps, it compensates for the blind spots of single-modality detection of subtle geometric defects, improving performance by 7.5% compared to the RGB-only method. Through intra-modal enhancement and cross-modal dynamic weighting, it fully exploits the complementarity of appearance and geometric information, improving accuracy by 3.1% compared to ordinary feature stitching. It requires no large pre-trained model or memory bank, achieving an inference speed of 20.91 FPS, and can be deployed on resource-constrained edge devices. On the MVTec 3D-AD dataset, it achieves an image-level AUROC of 92.9% and a pixel-level AUPRO of 95.6%, with a single image processing time of less than 50 milliseconds, meeting the real-time detection requirements of industry.
[0068] This invention achieves a good balance between detection accuracy, inference efficiency, and resource consumption, providing a high-precision, high-efficiency, and lightweight solution for product quality inspection in the industrial Internet of Things environment. Attached Figure Description
[0069] Figure 1 This is a schematic diagram of the overall process of the lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in this invention.
[0070] Figure 2This is a schematic diagram of the anomaly simulation generator in this invention;
[0071] Figure 3 This is a schematic diagram of the hierarchical cross-modal attention fusion module in this invention;
[0072] Figure 4 This is an example of the detection performance of this invention on the MVTec 3D-AD dataset. Detailed Implementation
[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0074] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0075] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.
[0076] Example 1:
[0077] The following is combined with Figures 1-3 This embodiment describes a lightweight anomaly detection method for the Industrial Internet of Things (IIoT) based on anomaly simulation and hierarchical multimodal feature fusion. The overall process is as follows: Figure 1 As shown, it includes the following steps:
[0078] Step 1, Anomaly Simulation:
[0079] RGB images and depth maps of normal products collected by industrial IoT sensors are used as normal samples; such as Figure 2 As shown, a generative anomaly enhancement strategy is adopted to synthesize pseudo-anomaly samples in the foreground region of normal samples, and a multimodal training set containing normal samples and pseudo-anomaly samples is constructed.
[0080] Furthermore, the specific method for synthesizing pseudo-abnormal samples in the foreground region of normal samples using the generative anomaly enhancement strategy includes:
[0081] Generate anomaly location masks using Perlin noise. ;
[0082] Extracting grayscale texture images from natural images ;
[0083] set up To be from the interval Uniformly sampled random numbers, The preset anomaly simulation probability;
[0084] For depth map First extract the foreground mask. The abnormal location mask is combined with the foreground mask to obtain the foreground abnormality mask. ;
[0085] if Then, anomaly simulation is performed to generate a pseudo-anomaly depth map. :
[0086] ;
[0087] in, To be from the interval Opacity parameter for uniform sampling Represents the Hadamard product;
[0088] when hour, ;
[0089] For RGB images Using the same foreground mask Generate a pseudo-abnormal RGB image.
[0090] In this embodiment, Perlin noise is used to generate anomaly location masks. The specific method is as follows:
[0091] Generate an initial anomaly map using a Perlin noise generator Pixel value range Inside. After affine transformation, according to Binarize the absolute value and filter out values less than the threshold. The components are used to obtain the abnormal location mask. ;
[0092] In this embodiment, .
[0093] Step 2, Image Reconstruction:
[0094] A lightweight dual-branch reconstruction network was trained using a multimodal training set to perform defect-free reconstruction of RGB images and depth maps, resulting in a reconstructed image corresponding to the input image.
[0095] Furthermore, the lightweight dual-branch reconstruction network adopts the Swin-Unet network with Swin Transformer as the backbone to reconstruct the RGB image and the depth map respectively.
[0096] A spatially weighted loss function is introduced during the reconstruction process to enhance the modeling ability of foreground defect regions;
[0097] Total reconstruction losses obtained Represented as:
[0098] ;
[0099] in, Indicates the input image. Indicates the reconstructed image. This represents the loss balance hyperparameter. This represents the total number of pixels in the image. This represents the cumulative sum of pixel-level reconstruction losses in the background region. This represents the cumulative sum of pixel-level reconstruction losses in the foreground region. Represents structural similarity loss;
[0100] in:
[0101] ;
[0102] ;
[0103] This represents the set of pixels in an image that belong to the background region. This represents the set of pixels in the image that belong to the foreground region. Indicates the position index of the pixel. Indicates the pixel position of the input image. Pixel value at that location, Indicates the pixel location of the reconstructed image. Pixel value at that location, Indicates the pixel position The L2 loss between the input image and the reconstructed image is calculated.
[0104] Furthermore, the lightweight dual-branch reconstruction network described in step 2 is trained with the discriminator generative adversarial network;
[0105] The discriminator uses a PatchGAN structure to distinguish between real samples and reconstructed samples.
[0106] The overall loss function of a generative adversarial network is expressed as a weighted sum of the adversarial losses of the RGB branch and the depth branch:
[0107] ;
[0108] in, This represents the adversarial loss function of the generator. This represents the loss balance coefficient for the RGB branch. This represents the loss balance coefficient for deep branches. Represents the mathematical expectation. This indicates the discriminator output for the RGB branch. The discriminator output represents the depth branch. The generator output represents the RGB branch. The generator output represents the depth branch. Indicates the input sample;
[0109] ;
[0110] in, This represents the total loss function of the generator. Indicates the losses incurred during reconstruction;
[0111]
[0112] in, This represents the adversarial loss function of the discriminator. Represents the actual samples of the RGB branch. This represents a real sample representing a deep branch.
[0113] Furthermore, , .
[0114] Step 3, Feature Fusion:
[0115] like Figure 3 As shown, multi-level features are extracted from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, respectively. The features of each modality are then fused through a hierarchical cross-modal attention fusion module to generate a fused feature map.
[0116] Furthermore, the hierarchical cross-modal attention fusion module described in step 3 includes: a single-modal feature extraction unit, an intra-modal feature enhancement unit, a hierarchical fusion unit, and a cross-modal attention unit;
[0117] The single-modal feature extraction unit uses a pre-trained ResNet50 as the backbone network to extract the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, respectively. Figure 4 The output of the first four residual blocks of each input is used to obtain multi-level semantic features;
[0118] The intramodal feature enhancement unit employs a dual attention module (DAM) obtained by combining the channel attention module (CAM) and the spatial attention module (SAM) to enhance the features at each level of each modality. The enhanced features are as follows:
[0119] ;
[0120] in, This represents the feature map of any mode at a certain level. This represents the Dual Attention Module (DAM). This represents the feature map after enhancement by the dual attention module;
[0121] The hierarchical fusion unit first performs element-wise addition and channel-wise concatenation fusion on the original RGB and reconstructed RGB feature maps to obtain an RGB fused feature map; simultaneously, it performs the same fusion operation on the original depth and reconstructed depth feature maps to obtain a depth fused feature map.
[0122] ;
[0123] ;
[0124] ;
[0125] ;
[0126] in, This represents the feature map of the original RGB image after enhancement by the dual attention module. This represents the feature map of the reconstructed RGB image after enhancement by the dual attention module. This represents the intermediate fused feature map obtained by adding RGB modes. This represents the intermediate fused feature map obtained by concatenating RGB modes. This represents the feature map after the original depth map has been enhanced by the dual attention module. This represents the feature map after the reconstructed depth map has been enhanced by the dual attention module. This represents the intermediate fused feature map obtained by adding deep modes. This represents the intermediate fused feature map obtained by concatenating deep modes;
[0127] The cross-modal attention unit projects the RGB fused feature map and the deep fused feature map through convolutional layers, calculates their correlation matrix, normalizes it using softmax to obtain attention weights, and then performs a weighted sum of the RGB fused feature map and the deep fused feature map according to the weights to generate the final fused feature map.
[0128] ;
[0129] in, This represents the final output fused feature map. This means adding the weighted RGB features to the depth features element by element. This represents the cross-modal attention function.
[0130] Furthermore, the channel attention module performs average pooling and max pooling on the feature map along the channel dimension, concatenates the results, and generates spatial attention weights through a convolutional layer.
[0131] Step 4: Input the fused feature map into the segmentation network, perform pixel-level anomaly classification, and output the segmentation results of the anomaly region.
[0132] Furthermore, the segmentation network described in step 4 adopts a U-Net structure, increasing the number of skip connections in the encoder, introducing transposed convolutions and upsampling in the decoder, and adding dilated convolutional layers to expand the receptive field; the segmentation network is trained using a focal loss function, which is expressed as:
[0133] ;
[0134] in, This represents the actual label mask. This represents the output predicted probability map of the segmentation network. This represents the predicted probability that each pixel belongs to the true class. Represents the category balance coefficient. This indicates the focus parameter.
[0135] Furthermore, the lightweight dual-branch reconstruction network and segmentation network employ a joint optimization strategy, with optimization objectives including reconstruction loss and classification loss, and the total loss function being:
[0136] ;
[0137] in, Represents the total loss function. This represents the reconstruction loss of the RGB branch. This represents the reconstruction loss for deep branches. Indicates focal loss. This represents the loss balance hyperparameter.
[0138] In this embodiment, the core principle of the proposed anomaly detection method lies in addressing the scarcity of anomalous samples in industrial scenarios through generative anomaly simulation. It utilizes a lightweight bi-branch reconstruction network and a hierarchical cross-modal attention fusion mechanism to achieve efficient collaborative analysis of RGB images and depth maps. Specifically, the method first synthesizes pseudo-anomaly samples with diverse morphologies in the foreground region of normal samples based on the fusion of Perlin noise and natural textures. This allows the model to encounter rich anomaly patterns during training, thereby enhancing its generalization ability to real defects. Subsequently, a bi-branch reconstruction network based on the Swin Transformer is used to reconstruct the RGB image and depth map without defects. This network, by learning the distribution of normal data, can reconstruct a normal appearance even when the input contains anomalies, making the difference between the input and the reconstruction an effective basis for anomaly localization. Building upon this, a hierarchical cross-modal attention fusion module is introduced. Through intra-modal feature enhancement, inter-modal feature interaction, and cross-modal attention weighting, the complementarity between the appearance information of the RGB image and the geometric information of the depth map is fully explored to generate more discriminative fusion features. Finally, a segmentation network performs pixel-level classification based on the fusion features to achieve accurate localization of anomalous regions. The entire framework employs generative adversarial networks and joint loss optimization strategies, which maintains lightweight computational overhead while ensuring detection accuracy, making it suitable for resource-constrained industrial IoT edge deployment scenarios.
[0139] In this embodiment, experimental results on the MVTec 3D-AD dataset show that the image-level anomaly detection AUROC reaches 92.9%, achieving performance comparable to or even better than existing state-of-the-art methods across multiple categories. Compared to mainstream methods (such as M3DM and AST) that rely on large pre-trained models and feature memories, this method achieves faster inference speed (20.91 FPS) without using any external pre-trained models or feature memories, significantly reducing computational resource consumption and memory usage. Furthermore, validation on the EyeCandies-mini dataset demonstrates that this method also has strong detection capabilities for anomalies on complex textured surfaces. Ablation experiments further confirm that the proposed generative anomaly simulation strategy improves detection performance by approximately 1.7%, and the hierarchical cross-modal attention fusion module improves detection accuracy by 3.1% compared to ordinary feature stitching methods. In summary, this embodiment achieves a good balance between detection accuracy, inference efficiency, and resource consumption, making it particularly suitable for deployment on edge computing nodes in industrial IoT environments to achieve real-time, accurate, and automated detection of product surface defects.
[0140] Example 2:
[0141] The detection system described in this embodiment for implementing a lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion includes:
[0142] The data augmentation module is used to perform the anomaly simulation step. It takes the RGB image and depth map of normal products collected by industrial IoT sensors as input, and uses a generative anomaly augmentation strategy to synthesize pseudo-anomaly samples in the foreground region of normal samples to build a multimodal training set containing normal samples and pseudo-anomaly samples.
[0143] The image reconstruction module is used to perform image reconstruction steps. It uses a lightweight dual-branch reconstruction network to perform defect-free reconstruction of the RGB image and the depth map respectively, and obtains the reconstructed image corresponding to the input image.
[0144] The feature fusion module is used to perform the full feature fusion step, extracting multi-level features from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, and then fusing the features of each modality through the hierarchical cross-modal attention fusion module to generate a fused feature map;
[0145] The anomaly segmentation module is used to perform the anomaly localization step. It inputs the fused feature map into the segmentation network, performs pixel-level anomaly classification, and outputs the segmentation results of the anomaly region.
[0146] In this embodiment, the anomaly detection system is based on the complete technical process of the method described in Embodiment 1. It achieves end-to-end anomaly detection through the coordinated operation of four functional modules. The data augmentation module, as the system's front end, receives the original RGB images and depth maps collected by industrial IoT sensors. It synthesizes pseudo-anomaly samples online using a generative anomaly simulation strategy, constructing a multimodal training set rich in anomaly patterns, thus solving the model training difficulties caused by the scarcity of anomaly samples in actual production. The image reconstruction module, as the core processing unit of the system, uses a lightweight dual-branch reconstruction network to perform defect-free reconstruction of the input image. During training, this module learns the distribution characteristics of normal data through a generative adversarial mechanism and a spatially weighted loss function. In the inference phase, the difference between the reconstructed image output by this module and the original input constitutes the basic signal for anomaly detection. The feature fusion module further performs deep fusion of multi-level features of the original and reconstructed images: first, it enhances the feature representation within each modality through a dual attention mechanism; then, it dynamically calculates the correlation between RGB and depth features through a cross-modal attention unit to generate a fused feature map. This module fully utilizes the complementary advantages of RGB and depth information, making subsequent anomaly localization more accurate. The anomaly segmentation module, serving as the system's output, performs pixel-level classification of the fused feature map based on the U-Net structure, outputting the segmentation results of anomaly regions. All four modules undergo end-to-end training using a joint optimization strategy to ensure optimal overall performance.
[0147] In this invention, such as Figure 4 The figure shows an example of the detection performance of this invention on the MVTec 3D-AD dataset. As can be seen from the figure, for inputs containing defects (such as cable joints, pins, etc.), the reconstruction network can accurately restore the defective area to a normal appearance. The anomaly localization map output by the segmentation network highly overlaps with the real defective area, verifying the effectiveness and robustness of this invention.
[0148] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. A lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion, characterized in that, It includes the following steps: Step 1: Use the RGB image and depth map of a normal product collected by an industrial IoT sensor as a normal sample; A generative anomaly enhancement strategy is adopted to synthesize pseudo-anomaly samples in the foreground region of normal samples, and a multimodal training set containing normal samples and pseudo-anomaly samples is constructed. Step 2: Train a lightweight dual-branch reconstruction network using a multimodal training set to perform defect-free reconstruction of the RGB image and depth map respectively, and obtain the reconstructed image corresponding to the input image. Step 3: Extract multi-level features from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map respectively, and fuse the features of each modality through the hierarchical cross-modal attention fusion module to generate a fused feature map; Step 4: Input the fused feature map into the segmentation network to perform pixel-level anomaly classification and output the segmentation results of the anomaly region.
2. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 1, characterized in that, Step 1 describes a generative anomaly enhancement strategy to synthesize pseudo-anomaly samples in the foreground region of normal samples. The specific method includes: Generate anomaly location masks using Perlin noise. ; Extracting grayscale texture images from natural images ; set up To be from the interval Uniformly sampled random numbers, The preset anomaly simulation probability; For depth map First extract the foreground mask. The abnormal location mask is combined with the foreground mask to obtain the foreground abnormality mask. ; if Then, anomaly simulation is performed to generate a pseudo-anomaly depth map. : ； in, To be from the interval Opacity parameter for uniform sampling Represents the Hadamard product; when hour, ; For RGB images Using the same foreground mask Generate a pseudo-abnormal RGB image.
3. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 1, characterized in that, The lightweight dual-branch reconstruction network described in step 2 uses the Swin-Unet network with SwinTransformer as the backbone to reconstruct the RGB image and the depth map respectively. A spatially weighted loss function is introduced during the reconstruction process to enhance the modeling ability of foreground defect regions; Total reconstruction losses obtained Represented as: ； in, Indicates the input image. Indicates the reconstructed image. This represents the loss balance hyperparameter. This represents the total number of pixels in the image. This represents the cumulative sum of pixel-level reconstruction losses in the background region. This represents the cumulative sum of pixel-level reconstruction losses in the foreground region. Represents structural similarity loss; in: ； ； This represents the set of pixels in an image that belong to the background region. This represents the set of pixels in the image that belong to the foreground region. Indicates the position index of the pixel. Indicates the pixel position of the input image. Pixel value at that location, Indicates the pixel location of the reconstructed image. Pixel value at that location, Indicates the pixel position The L2 loss between the input image and the reconstructed image is calculated.
4. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 1, characterized in that, The lightweight dual-branch reconstruction network and the discriminator generative adversarial network described in step 2 are trained together. The discriminator uses a PatchGAN structure to distinguish between real samples and reconstructed samples. The overall loss function of a generative adversarial network is expressed as a weighted sum of the adversarial losses of the RGB branch and the depth branch: ； in, This represents the adversarial loss function of the generator. This represents the loss balance coefficient for the RGB branch. This represents the loss balance coefficient for deep branches. Represents the mathematical expectation. This indicates the discriminator output for the RGB branch. The discriminator output represents the depth branch. The generator output represents the RGB branch. The generator output represents the depth branch. Indicates the input sample; ； in, This represents the total loss function of the generator. Indicates the losses incurred during reconstruction; in, This represents the adversarial loss function of the discriminator. Represents the actual samples of the RGB branch. This represents a real sample representing a deep branch.
5. A lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 4, characterized in that, The , .
6. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 1, characterized in that, The hierarchical cross-modal attention fusion module described in step 3 includes: a single-modal feature extraction unit, an intra-modal feature enhancement unit, a hierarchical fusion unit, and a cross-modal attention unit; The single-modal feature extraction unit uses a pre-trained ResNet50 as the backbone network to extract the outputs of the first four residual blocks of the four inputs: the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, respectively, to obtain multi-level semantic features. The intramodal feature enhancement unit employs a dual attention module (DAM) obtained by combining the channel attention module (CAM) and the spatial attention module (SAM) to enhance the features at each level of each modality. The enhanced features are as follows: ； in, This represents the feature map of any mode at a certain level. This represents the Dual Attention Module (DAM). This represents the feature map after enhancement by the dual attention module; The hierarchical fusion unit first performs element-wise addition and channel-wise concatenation fusion on the original RGB and reconstructed RGB feature maps to obtain an RGB fused feature map; simultaneously, it performs the same fusion operation on the original depth and reconstructed depth feature maps to obtain a depth fused feature map. ； ； ； ； in, This represents the feature map of the original RGB image after enhancement by the dual attention module. This represents the feature map of the reconstructed RGB image after enhancement by the dual attention module. This represents the intermediate fused feature map obtained by adding RGB modes. This represents the intermediate fused feature map obtained by concatenating RGB modes. This represents the feature map after the original depth map has been enhanced by the dual attention module. This represents the feature map after the reconstructed depth map has been enhanced by the dual attention module. This represents the intermediate fused feature map obtained by adding deep modes. This represents the intermediate fused feature map obtained by concatenating deep modes; The cross-modal attention unit projects the RGB fused feature map and the deep fused feature map through convolutional layers, calculates their correlation matrix, normalizes it using softmax to obtain attention weights, and then performs a weighted sum of the RGB fused feature map and the deep fused feature map according to the weights to generate the final fused feature map. ； in, This represents the final output fused feature map. This means adding the weighted RGB features to the depth features element by element. This represents the cross-modal attention function.
7. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 6, characterized in that, The channel attention module performs average pooling and max pooling on the feature map along the channel dimension, concatenates the results, and generates spatial attention weights through a convolutional layer.
8. The lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 1, characterized in that, The segmentation network described in step 4 adopts a U-Net structure, increasing the number of skip connections in the encoder, introducing transposed convolutions and upsampling in the decoder, and adding dilated convolutional layers to expand the receptive field; the segmentation network is trained using a focal loss function, which is expressed as: ； in, This represents the actual label mask. This represents the output predicted probability map of the segmentation network. This represents the predicted probability that each pixel belongs to the true class. Represents the category balance coefficient. This indicates the focus parameter.
9. A lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claim 8, characterized in that, The lightweight dual-branch reconstruction network and segmentation network employ a joint optimization strategy, with optimization objectives including reconstruction loss and classification loss. The total loss function is: ； in, Represents the total loss function. This represents the reconstruction loss of the RGB branch. This represents the reconstruction loss for deep branches. Indicates focal loss. This represents the loss balance hyperparameter.
10. A detection system for implementing the lightweight anomaly detection method for industrial IoT based on anomaly simulation and hierarchical multimodal feature fusion as described in claims 1-9, characterized in that, It includes: The data augmentation module is used to perform the anomaly simulation step. It takes the RGB image and depth map of normal products collected by industrial IoT sensors as input, and uses a generative anomaly augmentation strategy to synthesize pseudo-anomaly samples in the foreground region of normal samples to build a multimodal training set containing normal samples and pseudo-anomaly samples. The image reconstruction module is used to perform image reconstruction steps. It uses a lightweight dual-branch reconstruction network to perform defect-free reconstruction of the RGB image and the depth map respectively, and obtains the reconstructed image corresponding to the input image. The feature fusion module is used to perform the full feature fusion step, extracting multi-level features from the original RGB image, the reconstructed RGB image, the original depth map, and the reconstructed depth map, and then fusing the features of each modality through the hierarchical cross-modal attention fusion module to generate a fused feature map; The anomaly segmentation module is used to perform the anomaly localization step. It inputs the fused feature map into the segmentation network, performs pixel-level anomaly classification, and outputs the segmentation results of the anomaly region.