Multimodal industrial anomaly detection method integrating image mask and mixed experts

Abstract

The present invention discloses a multimodal industrial anomaly detection method that integrates image masking and hybrid experts. This method integrates image masking technology and hybrid expert methods to guide the industrial multimodal pre-training model. Based on the multimodal industrial data pre-training model, the pre-trained large model can learn features that are difficult to express with a single modality. During the training phase, masking technology is used to randomly mask part of the pixel blocks of the multimodal input image. Through training, the model is able to reconstruct the image-text aligned visual features of the masked image blocks based on the visible image blocks. The model can simultaneously learn low-level geometric structural features and high-level semantic information of the image, thereby covering most of the visual information required for downstream tasks. The encoder part of the large industrial model uses a hybrid expert structure instead of the forward propagation layer, which can dynamically activate a small number of network layers when processing data, thereby achieving the goal of accelerating model training and inference while maintaining the number of model parameters.

Description

Technical Field

[0001] The present invention relates to the field of industrial anomaly detection, and in particular to a multimodal industrial anomaly detection method integrating image mask and hybrid experts. Background Art

[0002] In recent years, large models have been able to effectively migrate to various downstream tasks and maintain good performance by pre-training on large amounts of data. Masked image modeling (MIM) performs a masking operation on the input image and, through training, enables the model to reconstruct the masked portion based on the visible image patches. However, using raw pixels as the reconstruction target has obvious limitations. Since natural raw images are information-sparse, simple pixel-level restoration tasks can often only capture low-level geometric and structural information while ignoring high-level semantic features. Therefore, it is difficult to maintain good performance with a billion-scale model size. The pre-training target is to reconstruct the image-text aligned visual features of the masked image patches. This pre-training task can simultaneously capture the geometric structural characteristics of natural images as well as high-level semantic abstract features, covering the information required for most visual tasks and achieving good performance in a wide range of downstream tasks.

[0003] As large models grow in size and complexity, with parameters scaling to billions or even trillions, maintaining efficiency and accuracy becomes a challenge. Traditional models activate all layers and neurons in the network for every input, often resulting in significant computational costs and memory consumption. The mixture of experts (MoE) model is an advanced network architecture that dynamically selects and activates experts to process different inputs during training and inference, reducing computational overhead while maintaining model performance. The MoE consists of a gating network and multiple expert networks. The gating network acts as a selector, sparsely activating a small number of experts for each input to process the data.

[0004] Traditional multimodal methods typically achieve 2D and 3D fusion through projection or feature concatenation. For example, they project a point cloud onto the image plane to generate a depth map, or combine features from both modalities through early / late fusion strategies. However, these methods often rely on precise sensor calibration and struggle to model deep semantic relationships between modalities. Furthermore, multimodal data in industrial scenarios often suffers from noise, occlusion, and sparsity (e.g., missing parts of point cloud structure), further complicating fusion. Furthermore, most current multimodal anomaly detection models cannot effectively handle the problem of missing modalities, or their performance degrades significantly when a modality is missing. Summary of the Invention

[0005] This paper aims to integrate image masks and hybrid expert models to propose a novel pre-training method for large industrial multimodal models. This method uses an image mask training strategy, with the pre-training goal being to reconstruct the CLIP visual features of the masked image. This method, based on self-supervised training rather than expensive supervised training, scales up the pre-trained large model and can lead to a qualitative improvement in learning performance during transfer, enabling large models to be better transferred to downstream tasks.

[0006] The object of the present invention is achieved through the following technical solution: a multimodal industrial anomaly detection method integrating image mask and hybrid expert, comprising the following steps:

[0007] S1, collect high-resolution aligned 2D images and 3D point cloud data of industrial components and perform preprocessing;

[0008] S2, projecting the 3D point cloud data onto the 2D image plane to construct a depth map, and projecting the depth map to a dimension aligned with the 2D image;

[0009] S3, dividing and randomly masking the 2D image and the projected 3D image;

[0010] S4. Build a large multimodal industrial anomaly detection model, comprising a linear projection layer for 3D data, an embedding layer, a ViT encoder module including a mixture of experts layer, and an MLP output head; and pre-train the CLIP features of the masked image patches as reconstruction targets.

[0011] S5. Freeze the linear projection layer of the 3D data, fine-tune the model, and use the pre-trained and fine-tuned model for industrial anomaly detection.

[0012] Furthermore, the acquisition of high-resolution aligned 2D images and 3D point cloud data of industrial components specifically includes: adding multimodal data of abnormal industrial components with various abnormal manifestations based on artificial simulation of defects, and acquiring paired 2D images and 3D point cloud data of the same object.

[0013] Furthermore, the preprocessing includes: performing dedistortion and normalization operations on the 2D image, correcting lens distortion according to the camera intrinsic parameters, removing outliers based on the DBSCAN algorithm for the 3D point cloud data, and adding Gaussian noise.

[0014] Furthermore, the projecting of 3D point cloud data onto a 2D image plane to construct a depth map includes: using the camera intrinsic parameter K and extrinsic parameter (R, t) to project the point cloud data onto a 2D image plane to construct a depth map. Projection onto a plane:

[0015]

[0016] According to the projection coordinates and depth value Generate depth map .

[0017] Furthermore, dividing and randomly masking the 2D image and the projected 3D image includes: inputting each modality into the image Divided into image blocks, where is the image resolution, is the number of channels, and the size of each image block is ;

[0018] The obtained multimodal image blocks are randomly sampled, and the input image blocks are destroyed with mask marks, keeping the average mask rate of the two-modal image blocks at 40%, and the unmasked image blocks are spliced.

[0019] Furthermore, the reconstruction goal is specifically to stitch the divided 2D image and the projected 3D image block, and use a CLIP encoder to extract image features, which serve as the mask reconstruction target for the subsequent pre-training task.

[0020] Furthermore, the multimodal industrial anomaly detection large model specifically includes: a linear projection layer for 3D data, an embedding layer, several ViT encoder layers and an MLP output head, wherein the unmasked visible image blocks are mapped to a D-dimensional space through the linear projection layer and position embedding is added, and then input into the ViT encoder layer, wherein the VIT encoder module includes a multi-head attention layer and a mixed expert layer, each mixed expert layer includes a gating network and several expert layers, and the obtained output is projected to the same dimension as the CLIP visual feature through the MLP output head.

[0021] Furthermore, the pre-training specifically includes: using CLIP visual features as reconstruction targets, calculating reconstruction loss using negative cosine similarity and performing back propagation, and the loss function is expressed as:

[0022]

[0023] in, is the visual feature output by the CLIP encoder, is the output of the multimodal industrial anomaly detection model, and N is the total number of image patches.

[0024] On the other hand, a multimodal industrial anomaly detection device that fuses image masks and mixed experts is also provided, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the multimodal industrial anomaly detection method that fuses image masks and mixed experts is implemented.

[0025] On the other hand, a computer-readable storage medium is also provided, on which a program is stored. When the program is executed by a processor, the multimodal industrial anomaly detection method that integrates image masks and mixed experts is implemented.

[0026] Beneficial effects of the present invention: Compared with the single-modal anomaly detection model, the multi-modal model can effectively utilize the characteristics of different modalities to make up for the shortcomings of other modalities. For example, the 2D image modality is more sensitive to the color attributes of the object, and can capture all the information on the surface of the object, but lacks the depth attribute; similarly, the 3D point cloud can effectively express the depth information of the object, but cannot perceive the internal information of the object. The multi-modal industrial anomaly detection large model described in the present invention will have a better anomaly detection effect than the single-modal model. At the same time, in actual detection, the data processed by the model described in the present invention will not be limited to single-modal or multi-modal input data, can effectively deal with the problem of missing modalities, and has rich application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the multimodal industrial large model anomaly detection method integrating image mask and hybrid experts implemented by the present invention;

[0028] Figure 2 This is a flow chart of the multimodal model pre-training method based on image mask reconstruction according to the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of the multimodal industrial anomaly detection large model of the present invention;

[0030] Figure 4 is a schematic diagram of the hybrid expert layer structure in the present invention;

[0031] Figure 5 An embodiment of the present invention provides a multimodal industrial anomaly detection device that integrates image masks and mixed experts. DETAILED DESCRIPTION

[0032] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0033] A multimodal industrial anomaly detection method integrating image mask and hybrid experts, such as Figure 1 The specific steps are as follows:

[0034] (1) Based on high-definition industrial cameras, high-precision millimeter-wave radars, lidars and other equipment, high-resolution aligned 2D images and 3D point cloud data of industrial components are collected, and 2D and 3D multimodal data of abnormal industrial components are added by artificially simulating defects. Data preprocessing involves cleaning the obtained industrial multimodal data sets, removing noise and low-quality data, and normalizing and standardizing the data to facilitate subsequent model processing.

[0035] (2) 3D point cloud processing: the input is a pair of aligned 2D images and 3D point cloud data, which are projected into a depth map using the calibrated camera and radar parameters. After a linear projection layer, the depth map obtained by projecting the 3D point cloud is reshaped to be consistent with the 2D image, and the two modal data are aligned in the same coordinate system.

[0036] (3) Image segmentation and masking: The 2D image and the projected 3D image are divided into image blocks of size 16×16. Some image blocks are randomly masked in proportion, that is, the content of the corresponding image blocks is destroyed. The mask reconstruction pre-training task aims to reconstruct the CLIP features of the masked image blocks.

[0037] (4) Model pre-training. This paper uses image mask reconstruction as a pre-training task, and the reconstruction target is CLIP visual features.

[0038] (5) Model fine-tuning and anomaly detection task application. The multimodal industrial anomaly model proposed in this invention can process single-modal data sets, such as pure image data or pure 3D point cloud data, and can also process multimodal data.

[0039] The multimodal data obtained in step (1) needs to be paired 2D images and 3D point cloud data based on the same object. The data set is required to meet the ratio of normal samples to abnormal samples of not less than 10:1. The multimodal data of abnormal industrial components added by artificial simulation of defects, etc., must include various forms of abnormalities such as scratches, dents, cracks, and surface damage.

[0040] In step (1), the data preprocessing operation is to dedistort and normalize the 2D image, and correct the lens distortion according to the camera intrinsic parameters. For the 3D point cloud data, outliers are removed based on the DBSCAN algorithm and appropriate Gaussian noise is added.

[0041] In step (2), the projection operation of the 3D point cloud data is performed by using the camera internal parameters (K) and external parameters (R, t) to transform the point cloud into Projection onto a plane:

[0042]

[0043] According to the projection coordinates and depth value Generate depth map .

[0044] In step (2), a linear projection layer is used to transform the depth map Projected to a dimension aligned with the 2D image, the same image segmentation method and Transformer encoder can be used to extract features of both modalities.

[0045] In step (3), for the aligned multimodal data, i.e., 3D point cloud image blocks and 2D image blocks, each modality input image Divided into image blocks, where is the image resolution, is the number of channels, and the size of each image block is The divided multimodal image blocks are spliced together, and a CLIP encoder is used to extract image features, which are used as the mask reconstruction target for subsequent pre-training tasks.

[0046] Regarding the CLIP encoder described in step (3), CLIP (Contrastive Language-Image Pre-Training) is a contrastive text-image pre-training model. The model is pre-trained with paired text-image data as input. It includes an encoder for image training and an encoder for text training, both of which use the Transformer encoder structure. The model learns the matching relationship between text-image pairs during the training phase. CLIP can effectively extract image-text alignment visual features, and the extracted image-text alignment visual features are the target of image mask reconstruction. The image features output by the CLIP encoder are ,in Image encoder for CLIP.

[0047] Regarding the image segmentation operation in step (3), the obtained multimodal image blocks are randomly sampled, and the input image blocks are destroyed with mask marks, keeping the average mask rate of the two-modal image blocks at 40%, and the masked image blocks are spliced.

[0048] Regarding the model pre-training task process in step (4) Figure 2 As shown, the left side of the figure shows data preprocessing and image segmentation operations. The pre-training process will continuously train the weights of the linear projection layer for processing 3D data. The multimodal image blocks obtained after image segmentation and image masking operations are subjected to feature extraction by the industrial anomaly detection large model described in the present invention. A linear projection layer is used to project the obtained multimodal features to the same dimension as the CLIP visual features. The goal of pre-training is to reconstruct the multimodal features into CLIP visual features, using negative cosine similarity as the mask reconstruction loss function. This training process can effectively improve the model's ability to extract deep-level image features. During the pre-training process, the relevant parameters of the frozen CLIP encoder will be frozen. Based on CLIP's excellent feature extraction capabilities, the parameters of the linear projection layer of 3D data, the anomaly detection large model, and the terminal linear projection are guided to update.

[0049] Regarding the multimodal industrial anomaly detection model mentioned in step (4), its structure is as follows Figure 3As shown, it includes a linear projection layer of 3D data, an embedding layer, N Vision Transformer (ViT) encoder modules, and an MLP output head.

[0050] Regarding the image embedding module in step (4), the image block is mapped to the D-dimensional space through the linear projection layer, and the position embedding is added to obtain the sequence ,in is the vector of the i-th image block, E is the projection matrix, Embed information for the position of the i-th image block, is the modality information of the i-th image block, which is input to the ViT encoder.

[0051] The output features of the visible image block after being processed by the ViT encoder. In the N-layer encoder, the shallow encoder focuses on extracting local features of the image data and is sensitive to position information. The high-level encoder focuses more on extracting global features of the image and has better robustness to local disturbances (such as noise and occlusion in the image). The multi-layer encoder design enables the model to locate anomalies more accurately. After being projected to the same dimension as the CLIP visual feature through the linear projection layer, ,in is the output feature after encoder processing, is the projection matrix.

[0052] The CLIP visual feature is used as the reconstruction target, and the reconstruction loss is calculated using negative cosine similarity, which is expressed as:

[0053]

[0054] The structure of the Mixture of Experts (MoE) in the ViT encoder structure in the industrial multimodal anomaly detection model in step (4) is as follows: Figure 4 As shown in Figure 1, MoE is a machine learning model that activates only part of the network when processing input data, thereby accelerating training and inference. In ViT's encoder architecture, the input sequence passes through each multi-head attention layer and FFN layer. MoE is introduced to replace the FFN layer. MoE consists of one gating network and n expert networks.

[0055] The hybrid expert neural network used in the present invention specifically includes the following expert networks:

[0056] 1 Fully Connected Expert: It includes two fully connected layers, and its mathematical expression is:

[0057]

[0058] Among them, the input of each expert network is x, 、 and 、 are the weight matrices and biases of the two fully connected layers.

[0059] 2 Convolutional Experts: includes one or more convolutional layers, and its mathematical expression is:

[0060]

[0061] Among them, Conv is the convolution function, K is the convolution kernel size, and b is the convolution layer bias.

[0062] 3. Zero Expert: Directly discard the input and output a zero vector. Its mathematical expression is:

[0063]

[0064] 4 Copy Expert: Skip the current expert and return to the input directly. Its mathematical expression is:

[0065]

[0066] 5 Constant Expert: Performs constant correction and replacement on input. Its mathematical expression is:

[0067] or

[0068] The gating network of the MoE layer is responsible for selecting a sparse expert combination for each input token. For input x, the output of the MoE layer is ,in is the output of the gating network, is the output of the i-th expert network. The gated network is a network with softmax. ,in is the weight matrix of the gating network for each expert, and the obtained The scores of each expert.

[0069] Experts are selected based on Top-K gating, where the k experts with the highest scores are retained and the others are set to 0.

[0070]

[0071] Based on this, , only the top k experts with the highest scores are retained to process the input data, and the remaining experts are not calculated, which can effectively reduce computing resources.

[0072] Regarding the model fine-tuning mentioned in step (5), after the pre-training task is completed, the multimodal anomaly model of the present invention has good generalization capabilities for processing multimodal industrial data. When processing different industrial datasets in actual applications, only fine-tuning is required to achieve good performance. During model fine-tuning, the parameters of the image embedding module, the MoE layer in the ViT encoder, and the forward propagation network are updated, while the 3D data projection module remains frozen.

[0073] Compared with single-modal anomaly detection models, multimodal models can effectively utilize the characteristics of different modalities and make up for the shortcomings of other modalities. For example, the 2D image modality is more sensitive to the color properties of objects and can capture all the information of the object surface, but lacks depth attributes; similarly, the 3D point cloud can effectively express the depth information of the object, but cannot perceive the internal information of the object.

[0074] Based on the multimodal industrial anomaly detection model described in the present invention, the backbone structure for multimodal data feature extraction is the same ViT encoder, so it can cope with the problem of modality missing. In actual detection, the processed data will not be limited to single-modal or multi-modal input data, and the application scenarios are rich.

[0075] Corresponding to the aforementioned embodiment of a multimodal industrial anomaly detection method that integrates image masks and mixed experts, the present invention also provides an embodiment of a multimodal industrial anomaly detection device that integrates image masks and mixed experts.

[0076] See also Figure 5 An embodiment of the present invention provides a multimodal industrial anomaly detection device that integrates image masks and mixed experts, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal industrial anomaly detection method that integrates image masks and mixed experts in the above embodiment.

[0077] An embodiment of a multimodal industrial anomaly detection device that integrates image masks and hybrid experts provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5As shown in the figure, a hardware structure diagram of a multimodal industrial anomaly detection device that integrates image mask and hybrid experts provided by the present invention is provided in any device with data processing capability, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0078] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0079] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0080] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for multimodal industrial anomaly detection by fusing image masks and hybrid experts in the above embodiment is implemented.

[0081] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0082] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the multimodal industrial anomaly detection method that integrates image masks and mixed experts.

[0083] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0084] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A multimodal industrial anomaly detection method integrating image mask and hybrid experts, characterized in that: The steps include: S1, collect high-resolution aligned 2D images and 3D point cloud data of industrial components and perform preprocessing; S2, projecting the 3D point cloud data onto the 2D image plane to construct a depth map, and projecting the depth map to a dimension aligned with the 2D image; S3, dividing and randomly masking the 2D image and the projected 3D image; S4. Build a large multimodal industrial anomaly detection model, comprising a linear projection layer for 3D data, an embedding layer, a ViT encoder module including a mixture of experts layer, and an MLP output head; and pre-train the CLIP features of the masked image patches as reconstruction targets. The multimodal industrial anomaly detection model specifically includes: a linear projection layer for 3D data, an embedding layer, several ViT encoder layers, and an MLP output head. The unmasked visible image blocks are mapped to a D-dimensional space through the linear projection layer and position embedding is added. The input is then fed into the ViT encoder layer. The VIT encoder module includes a multi-head attention layer and a mixture of experts layer. Each mixture of experts layer includes a gating network and several expert layers. The output is projected to the same dimension as the CLIP visual feature through the MLP output head. S5. Freeze the linear projection layer of the 3D data, fine-tune the model, and use the pre-trained and fine-tuned model for industrial anomaly detection.

2. A multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1, characterized in that: The acquisition of high-resolution aligned 2D images and 3D point cloud data of industrial components specifically includes: adding multimodal data of abnormal industrial components with various abnormal manifestations based on artificially simulated defects, and acquiring paired 2D images and 3D point cloud data for the same object.

3. The multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1 is characterized in that: The preprocessing includes: performing dedistortion and normalization operations on the 2D image, correcting the lens distortion according to the camera intrinsic parameters, removing outliers based on the DBSCAN algorithm for the 3D point cloud data, and adding Gaussian noise.

4. The multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1 is characterized in that: The projecting of 3D point cloud data onto a 2D image plane to construct a depth map includes: projecting the point cloud onto a plane using camera intrinsic parameters and extrinsic parameters, and generating a depth map according to the projection coordinates and the depth value of the point cloud.

5. The multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1 is characterized in that: The 2D image and the projected 3D image are divided and randomly masked, including: dividing the input image of each modality into a number of image blocks according to the size of the image block, randomly sampling the obtained multimodal image blocks, destroying the input image blocks with mask marks, maintaining the average mask rate of the two modal image blocks at 40%, and splicing the unmasked image blocks.

6. The multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1, characterized in that: The reconstruction goal is specifically to splice the divided 2D image and the projected 3D image block, and use a CLIP encoder to extract image features, which serve as the mask reconstruction target for subsequent pre-training tasks.

7. The multimodal industrial anomaly detection method integrating image mask and hybrid experts according to claim 1 is characterized in that: The pre-training specifically includes: using CLIP visual features as reconstruction targets, calculating reconstruction loss using negative cosine similarity based on the visual features output by the CLIP encoder and the output of the multimodal industrial anomaly detection large model, and performing backpropagation.

8. A multimodal industrial anomaly detection device integrating image mask and hybrid expert, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a multimodal industrial anomaly detection method integrating image mask and mixed experts is implemented as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a multimodal industrial anomaly detection method integrating image mask and mixed experts is implemented as described in any one of claims 1 to 7.

Images