An anomaly detection method for industrial large models by integrating image masks and hybrid experts

Abstract

The present invention discloses an industrial large-scale model anomaly detection method that integrates image masking and hybrid experts. This method integrates image masking technology and hybrid expert methods to guide industrial pre-training models. During the training phase, masking technology is used to mask part of the pixel blocks of the input image. Through training, the model can reconstruct the image-text aligned visual features of the masked image blocks based on the visible image blocks, so that the model can learn low-level geometric structural features and high-level semantic information of the image at the same time, and can cover most of the visual information required for downstream tasks. The encoder part of the pre-trained large model adopts a hybrid expert structure instead of the forward propagation layer, which can dynamically activate a small number of network layers when processing data, thereby greatly accelerating the model training and inference speed while maintaining the number of model parameters.

Description

Technical Field

[0001] The present invention relates to an anomaly detection method for industrial large models, and in particular to an anomaly detection method for industrial large models integrating image mask and hybrid experts. Background Art

[0002] In recent years, large models have been able to effectively migrate to various downstream tasks and maintain good performance by pre-training on large amounts of data. Masked image modeling (MIM) performs a masking operation on the input image and, through training, enables the model to reconstruct the masked portion based on the visible image patches. However, using raw pixels as the reconstruction target has obvious limitations. Since natural raw images are information-sparse, simple pixel-level restoration tasks can often only capture low-level geometric and structural information while ignoring high-level semantic features. Therefore, it is difficult to maintain good performance with a billion-scale model size. The pre-training target is to reconstruct the image-text aligned visual features of the masked image patches. This pre-training task can simultaneously capture the geometric structural characteristics of natural images as well as high-level semantic abstract features, covering the information required for most visual tasks and achieving good performance in a wide range of downstream tasks.

[0003] As large models grow in size and complexity, with parameters scaling to billions or even trillions, maintaining efficiency and accuracy becomes a challenge. Traditional models activate all layers and neurons in the network for every input, often resulting in significant computational costs and memory consumption. The mixture of experts (MoE) model is an advanced network architecture that dynamically selects and activates experts to process different inputs during training and inference, reducing computational overhead while maintaining model performance. The MoE consists of a gating network and multiple expert networks. The gating network acts as a selector, sparsely activating a small number of experts for each input to process the data. Summary of the Invention

[0004] This paper aims to propose a novel method for detecting anomalies in large industrial models by fusing image masks and hybrid expert models. This method uses an image mask training strategy to scale the model to billions of parameters based on large-scale unlabeled data. Through self-supervised training, it effectively detects anomalies in industrial images.

[0005] The object of the present invention is achieved through the following technical solution: a method for detecting anomalies in industrial large models by integrating image masks and hybrid experts, comprising the following steps:

[0006] S1. Obtain an industrial anomaly image dataset with manually labeled anomaly areas.

[0007] S2. Build a large-scale industrial anomaly detection model: The large-scale industrial anomaly detection model includes a position embedding, a VIT encoder module, and an MLP output head, where the VIT encoder includes a multi-head attention layer and a mixture of experts layer;

[0008] S3. Use the dataset to pre-train a large industrial anomaly detection model: Use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model. Project the output to the same dimension as the CLIP features and use negative cosine similarity as the loss function for training.

[0009] S4. Fine-tune the trained industrial anomaly detection model and perform anomaly detection on industrial images.

[0010] Furthermore, the acquisition of an industrial anomaly image dataset with manually labeled abnormal areas specifically includes: establishing an industrial anomaly detection benchmark image set by simulating or collecting real defective products in the production line as abnormal samples; preprocessing the industrial image dataset and manually labeling the abnormal areas of the abnormal images.

[0011] Furthermore, the positions in the industrial anomaly detection large model are embedded as a linear projection layer.

[0012] Furthermore, the VIT encoding layer in the industrial anomaly detection model is specifically:

[0013] The ViT encoder is composed of several encoder layers with the same structure. Each layer contains a multi-head attention mechanism and a mixed expert layer. The scaled dot product attention is used for calculation. The output of each layer of ViT adopts a residual structure. , where SubLayer represents a multi-head attention layer or a mixed expert layer;

[0014] The hybrid expert layer includes a gating network and several expert networks. The gating network uses TOP-K gating and is responsible for selecting a sparse expert combination for each input token.

[0015] Furthermore, the MLP output head takes the features extracted by the ViT encoder as input, and its structure consists of two fully connected layers. Relu is used as the activation function of the hidden layer, and the abnormal probability score is generated by the Sigmoid function to judge the abnormal information of the industrial image.

[0016] Furthermore, the CLIP encoder used in the pre-training process is comparative text-image pre-training, and the model is pre-trained using paired text-image data.

[0017] Furthermore, the fine-tuning is a two-stage training, including a hybrid expert layer training stage and a full network training stage; the hybrid expert layer training stage is the first stage, in which only the parameters of the gating network and the expert network in the hybrid expert layer are trained, and the weights of other structures in the block + position embedding module, MLP output head and Transformer encoder layer are frozen; the full network training stage is the second stage, in which all parameters will be trained.

[0018] Furthermore, the loss function is specifically:

[0019] The CLIP visual feature is used as the reconstruction target, and the reconstruction loss is calculated using negative cosine similarity, which is expressed as:

[0020]

[0021] in is the encoder output projected to the same dimension as the CLIP feature through the linear projection layer, is the image feature output by the CLIP encoder, and N is the total number of image blocks.

[0022] On the other hand, a device for detecting anomalies of industrial large models by fusing image masks and hybrid experts is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the method for detecting anomalies of industrial large models by fusing image masks and hybrid experts is implemented.

[0023] On the other hand, a computer-readable storage medium is also provided, on which a program is stored. When the program is executed by a processor, the method for detecting anomalies of industrial large models by fusing image masks and hybrid experts is implemented.

[0024] Beneficial effects:

[0025] The present invention integrates image masking and hybrid expert methods to achieve large-scale model pre-training on a large amount of unlabeled industrial data, and expands the model parameters to the billion level. The image masking strategy enables the model to maintain good performance when migrating to various industrial downstream fields while expanding the model parameters. At the same time, the forward propagation layer of the encoder part of the model is replaced by the hybrid expert method. Only a few expert networks are dynamically activated for each input data, which improves the speed and reduces the computing resource usage during large-scale model training and inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the industrial large model anomaly detection method integrating image mask and hybrid experts implemented by the present invention;

[0027] Figure 2This is a schematic diagram of the structure of the large-scale industrial anomaly detection model of the present invention;

[0028] Figure 3 This is a flow chart of the model pre-training method based on image mask reconstruction according to the present invention;

[0029] Figure 4 is a detailed schematic diagram of the hybrid expert layer in the present invention;

[0030] Figure 5 Schematic diagram of the two-stage training used for fine-tuning the model of the present invention;

[0031] Figure 6 A schematic diagram of an industrial large model anomaly detection device that integrates image masks and hybrid experts provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation method and working principle of the present invention are described in detail below with reference to the accompanying drawings:

[0033] An industrial large model anomaly detection method integrating image mask and mixed experts Figure 1 The specific steps are as follows:

[0034] S1. Use industrial high-definition cameras to capture product surface images under normal production conditions. Establish an industrial anomaly detection benchmark image set through simulations such as mechanical stress injection and artificial defects, or collect real defective products from the production line as anomaly samples. Preprocess the industrial image dataset using methods such as data cleaning and normalization, and manually annotate abnormal regions in abnormal images.

[0035] The industrial anomaly detection dataset must meet the requirement that the ratio of normal samples to abnormal samples is no less than 10:1. Real defect samples and abnormal samples generated by artificial simulation include scratches, dents, cracks, surface damage and other anomalies.

[0036] In the data preprocessing method, the normalization processing of the image data satisfies the mean μ=[0.485, 0.456, 0.406] and the standard deviation σ=[0.229, 0.224, 0.225].

[0037] S2. Establish a large model for industrial anomaly detection: The large model for industrial anomaly detection includes a position embedding, a VisionTransformer (ViT) encoder module, and an MLP output head.

[0038] When the large model detects industrial images, the input image block first passes through a linear projection layer to add position embedding information before serving as the input to the ViT encoder. The ViT encoder, based on a mixture of experts, includes a multi-head attention layer and a mixture of experts (MoE) layer. Each MoE layer contains a gating network and multiple types of expert networks. For each different input image block, the gating network sends the input data to a specific expert network, selectively activating some experts. After passing through multiple ViT encoders, the generated features are passed through an MLP output layer, whose output determines whether the image is abnormal and provides the model's predicted abnormal area.

[0039] The structure of the industrial anomaly detection model is as follows Figure 2 As shown, it includes an embedding layer, a Vision Transformer (ViT) encoder module, and an MLP output head. The embedding layer adds position embeddings to the input image blocks to preserve the spatial distribution information of the image blocks. The ViT encoder is composed of N = 10 encoder layers with the same structure. Each layer contains a multi-head attention mechanism with h = 12 heads. The scaled dot product attention is used for calculation: ,The mixed expert layer is used to replace the forward propagation layer in the traditional encoder structure, and each layer output of ViT adopts a residual structure , where SubLayer represents a multi-head attention layer or a mixed expert layer, and LayerNorm is a normalization layer. The MLP output head takes the features extracted by the ViT encoder as input. Its structure consists of two fully connected layers, using ReLU as the activation function of the hidden layer. The Sigmoid function generates anomaly probability scores to determine abnormal information in industrial images.

[0040] The hybrid expert layer is designed as Figure 4 As shown in the figure, a mixture of experts is a machine learning model that only activates specific expert networks when processing input data. This speeds up training and inference. In the ViT encoder architecture, the input sequence passes through the multi-head attention layer and the feed-forward network (FFN) layer at each layer. This paper introduces the MoE layer to replace the FFN layer. The MoE layer consists of one gating network and n expert networks.

[0041] The hybrid expert neural network used in the present invention specifically includes the following expert networks:

[0042] 1 Fully Connected Expert: It includes two fully connected layers, and its mathematical expression is:

[0043]

[0044] Among them, the input of each expert network is x, 、 and 、 are the weight matrices and biases of the two fully connected layers.

[0045] 2 Convolutional Experts: includes one or more convolutional layers, and its mathematical expression is:

[0046]

[0047] Among them, Conv is the convolution function, K is the convolution kernel size, and b is the convolution layer bias.

[0048] 3. Zero Expert: Directly discard the input and output a zero vector. Its mathematical expression is:

[0049]

[0050] 4 Copy Expert: Skip the current expert and return to the input directly. Its mathematical expression is:

[0051]

[0052] 5 Constant Expert: Performs constant correction and replacement on input. Its mathematical expression is:

[0053] or

[0054] The gating network of the MoE layer is responsible for selecting a sparse expert combination for each input token. For input x, the output of the MoE layer is ,in is the output of the gating network, is the output of the i-th expert network. The gated network is a network with Softmax. ,in is the weight matrix of the gating network for each expert, and the obtained The scores of each expert

[0055]

[0056] Where x is the input vector processed by the MoE layer each time, is the gating weight vector corresponding to the i-th expert, is the gating weight vector corresponding to the j-th expert.

[0057] Experts are selected based on Top-K gating, where the k experts with the highest scores are retained and the others are set to 0.

[0058]

[0059] Based on this, , only the top k experts with the highest scores are retained to process the input data, and the remaining experts are not calculated, which can effectively reduce computing resources.

[0060] The output y obtained after the MoE layer is connected to the input x residual to obtain the output of the Encoder .

[0061] S3. Use the dataset to pre-train a large industrial anomaly detection model: Use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model. Project the output to the same dimension as the CLIP features and use negative cosine similarity as the loss function for training.

[0062] During pre-training, the input image is first divided into multiple image blocks. Positional embedding information is then added to the image blocks through a linear projection layer. The CLIP encoder is used to extract image block features, generating CLIP visual features that serve as the image mask reconstruction targets during pre-training. The CLIP encoder parameters are frozen during training. The image blocks are randomly masked, and the unmasked visible image blocks serve as input to a large industrial anomaly detection model. Image features are extracted through a multi-layer ViT architecture, and the model's output features are projected to the same dimension as the CLIP features through a linear projection layer. During training, the model reconstructs the CLIP visual features of the masked image, using negative cosine similarity as the loss function to guide the mask reconstruction training process.

[0063] During the model fine-tuning and training process, a two-stage training method is adopted. In the first stage, only the MoE layer parameters are trained, and the remaining model weights are all frozen. In the second stage, all parameters are fine-tuned.

[0064] The pre-training process is as follows Figure 3 As shown, first input the image Divided into image blocks, where is the image resolution, is the number of channels, and the size of each image block is , and add embedding, which includes image block embedding and position embedding. The input image block is mapped to a D-dimensional vector (typical value D=768) through a learnable linear projection matrix. The position embedding uses a learnable absolute position encoding. The encoding dimension is the same as the image block embedding and is directly added and fused. After adding the embedding, the sequence is obtained ,in is the vector of the i-th image block, E is the projection matrix, Embed information for the position of the i-th image block.

[0065] The CLIP encoder, CLIP (Contrastive Language-Image Pre-Training) is a contrastive text-image pre-training model. The model uses paired text-image data for pre-training. It includes an encoder for image training and an encoder for text training, both of which use the Transformer encoder structure. The model learns the matching relationship between text-image pairs during the training phase. CLIP can effectively extract image-text alignment visual features. The extracted image-text alignment visual features are the target of image mask reconstruction. In the present invention, the visual encoder trained for images in CLIP is used to extract the visual features of industrial images. The image features output by the CLIP encoder are ,in Image encoder for CLIP.

[0066] The mask in step (2) randomly samples the obtained image blocks with a sampling rate of 40%, and the sampled image blocks are destroyed with mask marks. Only the visible image blocks are used as the input of the industrial large model. After being processed by the encoder, the visual features are projected to the same dimension as the CLIP features through a linear projection layer, which is expressed as

[0067] ,

[0068] in is the output feature after encoder processing, is the projection matrix of the linear layer.

[0069] The CLIP visual feature is used as the reconstruction target, and the reconstruction loss is calculated using negative cosine similarity, which is expressed as:

[0070]

[0071] S4. Fine-tune the trained industrial anomaly detection model and perform anomaly detection on industrial images. The two-stage training method used for model fine-tuning is as follows: Figure 5As shown in the figure, the left side is the first stage of training, in which only the parameters of the gating network and expert network in the MoE layer are trained, and the weights of other structures in the block + position embedding module, MLP output head and Transformer encoder layer are frozen; the right side is the second stage, in which all parameters will be trained.

[0072] In this paper, considering both model size and practical performance, the large industrial anomaly detection model used includes a 10-layer ViT encoder network. Each MoE layer includes four expert network layers. During training and inference, two experts are activated for each input, using a Top-2 gating strategy. The hybrid expert here applies a hybrid expert to the forward propagation of the ViT encoder network.

[0073] The above shows that the industrial large-scale model anomaly detection method that integrates image mask and hybrid expert proposed in the present invention can obtain an industrial large-scale model for abnormal image detection by pre-training on large-scale unlabeled industrial data. The model parameter scale in the present invention is 1.1B. Table 1 shows the anomaly detection performance comparison under the same parameter scale based on image mask, hybrid expert, and two-stage training. When pre-training based on image mask, the classification accuracy of anomaly detection is only 93.63%, while image mask pre-training can effectively enable the industrial large-scale model to learn abnormal information in a large amount of data, so that its anomaly classification accuracy reaches 95.12%. The hybrid expert structure added to the model further deepens the ability to process image information compared to the original pure ViT model, and the performance reaches 95.41%. Finally, the present invention introduces a two-stage training method in the fine-tuning stage. In the first stage, the model focuses on training the expert structure, so that the model further learns the feature extraction capability for the data based on the pre-training data. The second stage improves the robustness of the model, and finally achieves a detection accuracy of 96.93%.

[0074] Table 1 Comparison of anomaly detection performance of models based on the same parameters

[0075] Image mask pre-training Based on hybrid experts Based on two-stage training Anomaly classification accuracy × × × 93.63% √ × × 95.12% √ √ × 95.41% √ √ √ 96.93%

[0076] Corresponding to the aforementioned embodiment of an industrial large model anomaly detection method that integrates image mask and mixed experts, the present invention also provides an embodiment of an industrial large model anomaly detection device that integrates image mask and mixed experts.

[0077] See also Figure 6An embodiment of the present invention provides an industrial large model anomaly detection device that integrates image masks and mixed experts, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an industrial large model anomaly detection method that integrates image masks and mixed experts in the above embodiment.

[0078] The embodiment of the industrial large model anomaly detection device that integrates image masks and hybrid experts provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown, the hardware structure diagram of an industrial large model anomaly detection device that integrates image mask and hybrid experts provided by the present invention is a device with data processing capabilities. Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0079] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0081] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for detecting anomalies of an industrial large model by fusing image masks and hybrid experts in the above embodiment is implemented.

[0082] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0083] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the industrial large model anomaly detection method that integrates image masks and mixed experts.

[0084] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0085] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A method for detecting anomalies in industrial large models by integrating image masks and hybrid experts, characterized in that: The following steps are involved: S1. Obtain an industrial anomaly image dataset with manually labeled anomaly areas. S2. Build a large-scale industrial anomaly detection model: The large-scale industrial anomaly detection model includes a position embedding, a VIT encoder module, and an MLP output head, where the VIT encoder includes a multi-head attention layer and a mixture of experts layer; S3. Use the dataset to pre-train a large industrial anomaly detection model: Use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model. Project the output to the same dimension as the CLIP features and use negative cosine similarity as the loss function for training. S4. Fine-tune the trained large industrial anomaly detection model to perform anomaly detection on industrial images. The fine-tuning is a two-stage training, including a hybrid expert layer training stage and a full network training stage. The hybrid expert layer training stage is the first stage, in which only the parameters of the gating network and the expert network in the hybrid expert layer are trained, and the weights of other structures in the block + position embedding module, MLP output head, and Transformer encoder layer are frozen. The full network training stage is the second stage, in which all parameters will be trained.

2. The method for detecting anomalies in industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The method of obtaining an industrial anomaly image dataset with manually labeled abnormal areas specifically includes: establishing an industrial anomaly detection benchmark image set by simulating or collecting real defective products in a production line as abnormal samples; preprocessing the industrial image dataset and manually labeling abnormal areas of the abnormal images.

3. The method for detecting anomalies in industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The position embedding in the industrial anomaly detection large model is a linear projection layer.

4. The method for detecting anomalies in industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The VIT encoding layer in the industrial anomaly detection model is specifically: The ViT encoder is composed of several encoder layers with the same structure. Each layer contains a multi-head attention mechanism and a mixed expert layer. The scaled dot product attention is used for calculation. The output of each layer of ViT adopts a residual structure. , where SubLayer represents the multi-head attention layer or the mixed expert layer, and LayerNorm is the normalization layer; The hybrid expert layer includes a gating network and several expert networks. The gating network uses TOP-K gating and is responsible for selecting a sparse expert combination for each input token.

5. The method for detecting anomalies in industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The MLP output head takes the features extracted by the ViT encoder as input. Its structure consists of two fully connected layers, uses ReLU as the activation function of the hidden layer, and generates anomaly probability scores through the Sigmoid function to judge the abnormal information of industrial images.

6. The method for detecting anomalies in industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The CLIP encoder used in the pre-training process is a comparative text-image pre-training model, which uses paired text-image data for pre-training.

7. The method for detecting anomalies in industrial large models by fusing image masks and hybrid experts according to claim 1, characterized in that: The loss function is specifically as follows: using CLIP visual features as the reconstruction target, projecting the encoder output to the same dimension as the CLIP features through a linear projection layer, and the image features output by the CLIP encoder, and calculating the negative cosine similarity as the reconstruction loss.

8. An industrial large model anomaly detection device integrating image mask and hybrid expert, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the industrial large model anomaly detection method integrating image mask and hybrid expert according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an industrial large model anomaly detection method integrating image mask and hybrid expert is implemented as described in any one of claims 1 to 7.

Images