Industrial large model anomaly detection method fusing image masks and mixed experts

Abstract

The invention discloses an industrial large model anomaly detection method fusing image masks and mixed experts. According to the method, an image mask technology and a hybrid expert method are fused to guide an industrial pre-training model, part of pixel blocks of an input image are masked by using the mask technology in the training stage, and the model can reconstruct visual features aligned with image texts of masked image blocks according to visible image blocks through training, so that the image text alignment of the masked image blocks is realized. Therefore, the model learns low-level geometric structure features and high-level semantic information of the image at the same time, and visual information required by most downstream tasks can be covered. An encoder part of the pre-training large model adopts a hybrid expert structure to replace a forward propagation layer, and a small part of network layers can be dynamically activated during data processing, so that the model training and reasoning speed is greatly increased while the model parameter quantity is kept.

Description

Technical Field

[0001] The present invention relates to an industrial large model anomaly detection method, and in particular to an industrial large model anomaly detection method integrating image mask and mixed experts. Background Art

[0002] In recent years, large models can be effectively transferred to various downstream tasks and maintain good performance by pre-training on large amounts of data. Masked image modeling (MIM) performs masking operations on the input image, and through training, the model can reconstruct the masked part based on the visible image blocks. However, there are obvious limitations in using original pixels as the reconstruction target. Since natural original images are information-sparse, simple pixel-level restoration tasks can often only capture low-level geometric and structural information, while ignoring high-level semantic features. Therefore, it is difficult to maintain good performance under a billion-scale model size. The pre-training target is to reconstruct the image-text aligned visual features of the masked image blocks. This pre-training task can simultaneously capture the geometric structure information and high-level semantic abstract features of natural images, covering the information required for most visual tasks, and can achieve good performance in a wide range of downstream tasks.

[0003] As the parameters of large models become larger and their structures become more complex, the parameters expand to billions or even trillions of sizes, and how to maintain efficiency and accuracy becomes a challenge. Traditional models activate all layers and neurons in the network for each input, which often leads to huge computational costs and memory consumption. The mixture of experts (MoE) model is an advanced network architecture that dynamically selects and activates some experts to process different inputs during training and inference, which can reduce computational overhead while maintaining model performance. MoE includes a gating network and multiple expert networks. The gating network acts as a selector, sparsely activating a small number of experts for each input to process data. Summary of the invention

[0004] The purpose of this invention is to fuse image mask and hybrid expert model to propose a novel industrial large model anomaly detection method. This method uses image mask training strategy to expand the model to billions of parameters based on large-scale unlabeled data, and can effectively detect industrial abnormal images through self-supervised training.

[0005] The object of the present invention is achieved by the following technical solution: A method for detecting anomalies of industrial large models by integrating image masks and hybrid experts, comprising the following steps:

[0006] S1. Obtain an industrial anomaly image dataset with manually labeled anomaly areas.

[0007] S2. Establish a large model for industrial anomaly detection: The large model for industrial anomaly detection includes a position embedding, a VIT encoder module, and an MLP output head, wherein the VIT encoder includes a multi-head attention layer and a mixed expert layer;

[0008] S3. Use the dataset to pre-train the large industrial anomaly detection model: use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model, and project the output to the same dimension as the CLIP features, and use negative cosine similarity as the loss function for training;

[0009] S4. Fine-tune the trained industrial anomaly detection model and perform anomaly detection on industrial images.

[0010] Furthermore, the acquisition of an industrial anomaly image dataset with manually annotated abnormal areas specifically includes: establishing an industrial anomaly detection benchmark image set by simulating or collecting real defective products in a production line as abnormal samples; preprocessing the industrial image dataset and manually annotating abnormal areas of the abnormal images.

[0011] Furthermore, the positions in the industrial anomaly detection large model are embedded as a linear projection layer.

[0012] Furthermore, the VIT encoding layer in the industrial anomaly detection model is specifically:

[0013] The ViT encoder is composed of several encoder layers with the same structure. Each layer contains a multi-head attention mechanism and a mixed expert layer. The scaled dot product attention is used for calculation. The output of each layer of ViT adopts a residual structure. , where SubLayer represents a multi-head attention layer or a mixed expert layer;

[0014] The hybrid expert layer includes a gating network and several expert networks. The gating network uses TOP-K gating and is responsible for selecting a sparse expert combination for each input token.

[0015] Furthermore, the MLP output head takes the features extracted by the ViT encoder as input, and its structure is two fully connected layers. Relu is used as the activation function of the hidden layer, and the abnormal probability score is generated through the Sigmoid function to judge the abnormal information of the industrial image.

[0016] Furthermore, the CLIP encoder used in the pre-training process is comparative text-image pre-training, and the model is pre-trained using paired text-image data.

[0017] Furthermore, the fine-tuning is a two-stage training, including a hybrid expert layer training stage and a full network training stage; the hybrid expert layer training stage is the first stage, in which only the parameters of the gating network and the expert network in the hybrid expert layer are trained, and the weights of other structures in the block + position embedding module, the MLP output head and the Transformer encoder layer are frozen; the full network training stage is the second stage, in which all parameters will be trained.

[0018] Furthermore, the loss function is specifically:

[0019] The CLIP visual feature is used as the reconstruction target, and the reconstruction loss is calculated using negative cosine similarity, which is expressed as:

[0020]

[0021] in is the encoder output projected to the same dimension as the CLIP feature through the linear projection layer, is the image feature output by the CLIP encoder, and N is the total number of image blocks.

[0022] On the other hand, a device for detecting anomalies of industrial large models by fusing image masks and hybrid experts is provided, comprising a memory and one or more processors, wherein executable codes are stored in the memory, and when the processor executes the executable codes, the method for detecting anomalies of industrial large models by fusing image masks and hybrid experts is implemented.

[0023] On the other hand, a computer-readable storage medium is also provided, on which a program is stored. When the program is executed by a processor, the industrial large model anomaly detection method that integrates image masks and mixed experts is implemented.

[0024] Beneficial effects:

[0025] The present invention integrates image masking and hybrid expert methods to achieve large-scale model pre-training on a large amount of unlabeled industrial data, and expands the model parameters to the billion level. The image masking strategy is used to expand the model parameters while maintaining good performance when the model is migrated to various industrial downstream fields. At the same time, the encoder part of the model is replaced with the forward propagation layer using the hybrid expert method, and only a few expert networks are dynamically activated for each input data, thereby improving the speed of large-scale model training and reasoning and reducing the computing resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of the industrial large model anomaly detection method by fusing image mask and hybrid experts implemented by the present invention;

[0027] Figure 2This is a schematic diagram of the structure of the large industrial anomaly detection model of the present invention;

[0028] Figure 3 This is a flow chart of the model pre-training method based on image mask reconstruction according to the present invention;

[0029] Figure 4 is a detailed schematic diagram of the hybrid expert layer in the present invention;

[0030] Figure 5 A schematic diagram of the two-stage training used for fine-tuning the model of the present invention;

[0031] Figure 6 A schematic diagram of an industrial large model anomaly detection device integrating image mask and hybrid expert provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation method and working principle of the present invention are described in detail below in conjunction with the accompanying drawings:

[0033] An industrial large model anomaly detection method integrating image mask and hybrid experts Figure 1 As shown, the specific steps are as follows:

[0034] S1. Use industrial high-definition cameras to collect product surface images under normal production conditions, and establish an industrial anomaly detection benchmark image set through simulation methods such as mechanical stress injection and artificial defects, or collect real defective products in the production line as abnormal samples. Preprocess the industrial image data set, process the image data using methods such as data cleaning and normalization, and manually annotate the abnormal areas of abnormal images.

[0035] The industrial anomaly detection data set must meet the requirement that the ratio of normal samples to abnormal samples is no less than 10:1. The real defect samples and the abnormal samples generated by artificial simulation include scratches, dents, cracks, surface damage and other anomalies.

[0036] In the data preprocessing method, the normalization processing of the image data satisfies the mean μ=[0.485, 0.456, 0.406] and the standard deviation σ=[0.229, 0.224, 0.225].

[0037] S2. Establish a large model for industrial anomaly detection: The large model for industrial anomaly detection includes a position embedding, a VisionTransformer (ViT) encoder module, and an MLP output head.

[0038] When the large model detects industrial images, the input image block first passes through a linear projection layer to add position embedding information and then serves as the input of the ViT encoder. The ViT encoder based on hybrid experts includes a multi-head attention layer (Multi-HeadAttention) and a mixture of experts (Mixture of Experts, MoE). Each MoE layer contains a gating network and multiple types of expert networks. For each different input image block, the gating network sends the input data to a specific expert network and selectively activates some experts. After passing through multiple ViT encoders, the generated features will pass through an MLP output layer, whose output will determine whether the image is abnormal and give the abnormal area predicted by the model.

[0039] The structure of the industrial anomaly detection model is as follows Figure 2 As shown, it includes an embedding layer, a Vision Transformer (ViT) encoder module, and an MLP output head. The embedding layer adds position embedding to the input image block to retain the spatial distribution information of the image block. The ViT encoder is composed of N=10 encoder layers with the same structure. Each layer contains a multi-head attention mechanism with h=12 heads. The scaled dot product attention is used for calculation: ,The mixed expert layer replaces the forward propagation layer in the traditional encoder structure, and each layer output of ViT adopts a residual structure , where SubLayer represents a multi-head attention layer or a mixed expert layer, and LayerNorm is a normalization layer. The MLP output head takes the features extracted by the ViT encoder as input. Its structure is two fully connected layers, using Relu as the activation function of the hidden layer. The abnormal probability score is generated through the Sigmoid function to judge the abnormal information of the industrial image.

[0040] The hybrid expert layer is designed as Figure 4 As shown. Hybrid Expert is a machine learning model that only activates a specific expert network when processing input data, so as to speed up training and reasoning. In the encoder architecture of ViT, the input sequence will pass through the multi-head attention layer and the forward propagation layer (Feed-Forward Network, FFN) of each layer. The present invention introduces the MoE layer to replace the FFN layer. The MoE layer includes 1 gated network and n expert networks.

[0041] The hybrid expert neural network used in the present invention specifically includes the following expert networks:

[0042] 1 Fully Connected Expert: It includes two fully connected layers, and its mathematical expression is:

[0043]

[0044] Among them, the input of each expert network is x, , and , are the weight matrices and biases of the two fully connected layers.

[0045] 2Convolutional Experts: includes one or more convolutional layers, and its mathematical expression is:

[0046]

[0047] Among them, Conv is the convolution function, K is the convolution kernel size, and b is the convolution layer bias.

[0048] 3Zero Expert: Directly discard the input and output a zero vector. Its mathematical expression is:

[0049]

[0050] 4 Copy Expert: Skip the current expert and return to the input directly. Its mathematical expression is:

[0051]

[0052] 5 Constant Expert: Perform constant correction and replacement on input. Its mathematical expression is:

[0053] or

[0054] The gating network of the MoE layer is responsible for selecting a sparse expert combination for each input token. For input x, the output of the MoE layer is ,in is the output of the gating network, is the output of the i-th expert network. The gated network is a network with Softmax. ,in is the weight matrix of the gating network for each expert, and the obtained The scores of the experts

[0055]

[0056] Where x is the input vector processed by the MoE layer each time, is the gating weight vector corresponding to the i-th expert, is the gating weight vector corresponding to the j-th expert.

[0057] The selection of experts is based on Top-K gating, which selects the k experts with the highest scores and sets the others to 0.

[0058]

[0059] Based on this, , only the first k experts with the highest scores are retained to process the input data, and the remaining experts do not perform calculations, which can effectively reduce computing resources.

[0060] The output y obtained after the MoE layer is connected to the input x residual to get the output of the Encoder .

[0061] S3. Use the dataset to pre-train the large industrial anomaly detection model: use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model, and project the output to the same dimension as the CLIP features, and use negative cosine similarity as the loss function for training;

[0062] During the pre-training process, the input image is first divided into multiple image blocks, and position embedding information is added to the image blocks through a linear projection layer. The CLIP encoder is used to extract image block features and generate CLIP visual features as the image mask reconstruction target of the pre-training process. The parameters of the CLIP encoder are frozen during the training process. The image blocks are randomly masked, and the unmasked visible image blocks are used as the input of the large industrial anomaly detection model. Image features are extracted through a multi-layer ViT structure, and the output features of the model are projected to the same dimension as the CLIP features through a linear projection layer. The model reconstructs the CLIP visual features of the masked image during training, and uses negative cosine similarity as the loss function to guide the training process of mask reconstruction.

[0063] During the model fine-tuning and training process, a two-stage training method is adopted. In the first stage, only the MoE layer parameters are trained, and the remaining model weights are all frozen. In the second stage, all parameters are fine-tuned.

[0064] The pre-training process is as follows Figure 3 As shown, first input the image Divide into image blocks, where is the image resolution, is the number of channels, and the size of each image block is , and add embedding, which includes image block embedding and position embedding. The input image block is mapped to a D-dimensional vector (typical value D=768) through a learnable linear projection matrix. The position embedding uses a learnable absolute position encoding. The encoding dimension is the same as the image block embedding and is directly added and fused. After adding the embedding, the sequence is obtained ,in is the vector of the i-th image block, E is the projection matrix, Embed information for the position of the i-th image block.

[0065] The CLIP encoder, CLIP (Contrastive Language-Image Pre-Training) is contrastive text-image pre-training. The model uses paired text-image data for pre-training. It includes an encoder for image training and an encoder for text training. Both use the Transformer encoder structure. The model learns the matching relationship between text-image pairs during the training phase. CLIP can effectively extract image-text alignment visual features. The extracted image-text alignment visual features are the target of image mask reconstruction. In the present invention, the visual encoder trained for images in CLIP is used to extract the visual features of industrial images. The image features output by the CLIP encoder are ,in Image encoder for CLIP.

[0066] The mask in step (2) randomly samples the obtained image blocks with a sampling rate of 40%, and destroys the sampled image blocks with mask marks. Only the visible image blocks are used as the input of the industrial large model. After being processed by the encoder, the visual features are projected to the same dimension as the CLIP features through a linear projection layer, expressed as

[0067] ,

[0068] in is the output feature after being processed by the encoder, is the projection matrix of the linear layer.

[0069] The CLIP visual feature is used as the reconstruction target, and the reconstruction loss is calculated using negative cosine similarity, which is expressed as:

[0070]

[0071] S4. Fine-tune the trained industrial anomaly detection model to detect anomalies on industrial images. The two-stage training method used for model fine-tuning is as follows: Figure 5As shown in the figure, the left side is the first stage of training, in which only the parameters of the gating network and the expert network in the MoE layer are trained, and the weights of other structures in the block + position embedding module, the MLP output head, and the Transformer encoder layer are frozen; the right side is the second stage, in which all parameters will be trained.

[0072] In the present invention, considering the model size and actual performance, the industrial anomaly detection large model used includes a 10-layer ViT encoder network, each MoE layer includes 4 expert network layers, and the training and reasoning process activates 2 experts for each input, that is, a Top-2 gating strategy is adopted. The hybrid expert here is a hybrid expert applied to the forward propagation in the ViT encoder network.

[0073] The above shows that the industrial large model anomaly detection method that integrates image mask and hybrid expert proposed in the present invention can obtain an industrial large model for abnormal image detection through pre-training on large-scale unlabeled industrial data. The model parameter scale in the present invention is 1.1B. Table 1 is a comparison of anomaly detection performance under the same parameter scale based on image mask, hybrid expert, and two-stage training. When pre-training based on image mask, the classification accuracy of anomaly detection is only 93.63%, while image mask pre-training can effectively enable the industrial large model to learn abnormal information in a large amount of data, so that its abnormal classification accuracy reaches 95.12%. The hybrid expert structure added to the model further deepens the ability to process image information compared to the original pure ViT model, and the performance reaches 95.41%. Finally, the present invention introduces a two-stage training method in the fine-tuning stage. In the first stage, the model focuses on training the expert structure, so that the model can further learn the feature extraction ability for the data based on the pre-training data. The second stage improves the robustness of the model, and finally reaches a detection accuracy of 96.93%.

[0074] Table 1 Comparison of anomaly detection performance of models based on the same parameters

[0075] Image mask based pre-training Based on hybrid experts Based on two-stage training Anomaly classification accuracy × × × 93.63% √ × × 95.12% √ √ × 95.41% √ √ √ 96.93%

[0076] Corresponding to the aforementioned embodiment of an industrial large model anomaly detection method that integrates image mask and mixed experts, the present invention also provides an embodiment of an industrial large model anomaly detection device that integrates image mask and mixed experts.

[0077] See also Figure 6An embodiment of the present invention provides an industrial large model anomaly detection device that integrates image masks and mixed experts, including a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement an industrial large model anomaly detection method that integrates image masks and mixed experts in the above embodiment.

[0078] An embodiment of an industrial large model anomaly detection device that integrates image masks and hybrid experts provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown, a hardware structure diagram of an industrial large model anomaly detection device for integrating image mask and hybrid expert provided by the present invention is located in any device with data processing capability, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0079] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0080] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0081] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, an industrial large model anomaly detection method that fuses image masks and mixed experts in the above embodiment is implemented.

[0082] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0083] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the industrial large model anomaly detection method that integrates image masks and mixed experts.

[0084] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0085] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. An industrial large model anomaly detection method integrating image mask and hybrid experts, characterized in that: The following steps are involved: S1. Obtain an industrial anomaly image dataset with manually labeled anomaly areas. S2. Establish a large model for industrial anomaly detection: The large model for industrial anomaly detection includes a position embedding, a VIT encoder module, and an MLP output head, wherein the VIT encoder includes a multi-head attention layer and a mixed expert layer; S3. Use the dataset to pre-train the large industrial anomaly detection model: use the CLIP visual features extracted by the CLIP encoder as the image mask reconstruction target, randomly mask the image and use the unmasked visible image as the input of the industrial anomaly detection model, and project the output to the same dimension as the CLIP features, and use negative cosine similarity as the loss function for training; S4. Fine-tune the trained industrial anomaly detection model and perform anomaly detection on industrial images.

2. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The method of obtaining an industrial anomaly image dataset with manually annotated abnormal areas specifically includes: establishing an industrial anomaly detection benchmark image set by simulating or collecting real defective products in a production line as abnormal samples; preprocessing the industrial image dataset and manually annotating abnormal areas of the abnormal images.

3. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The position embedding in the industrial anomaly detection large model is a linear projection layer.

4. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The VIT encoding layer in the industrial anomaly detection model is specifically: The ViT encoder is composed of several encoder layers with the same structure. Each layer contains a multi-head attention mechanism and a mixed expert layer. The scaled dot product attention is used for calculation. The output of each layer of ViT adopts a residual structure. , where SubLayer represents a multi-head attention layer or a mixed expert layer, and LayerNorm is a normalization layer; The hybrid expert layer includes a gating network and several expert networks. The gating network uses TOP-K gating and is responsible for selecting a sparse expert combination for each input token.

5. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The MLP output head takes the features extracted by the ViT encoder as input, and its structure consists of two fully connected layers. Relu is used as the activation function of the hidden layer, and the abnormal probability score is generated by the Sigmoid function to judge the abnormal information of the industrial image.

6. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The CLIP encoder used in the pre-training process is a comparative text-image pre-training model, which is pre-trained using paired text-image data.

7. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The fine-tuning is a two-stage training, including a hybrid expert layer training stage and a full network training stage; the hybrid expert layer training stage is the first stage, in which only the parameters of the gating network and the expert network in the hybrid expert layer are trained, and the weights of other structures in the block + position embedding module, the MLP output head and the Transformer encoder layer are frozen; the full network training stage is the second stage, in which all parameters will be trained.

8. The method for detecting anomalies of industrial large models by integrating image mask and hybrid experts according to claim 1, characterized in that: The loss function is specifically as follows: taking the CLIP visual feature as the reconstruction target, projecting the encoder output to the same dimension as the CLIP feature through a linear projection layer, and the image feature output by the CLIP encoder, and calculating the negative cosine similarity as the reconstruction loss.

9. An industrial large model anomaly detection device integrating image mask and hybrid expert, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, an industrial large model anomaly detection method integrating image mask and hybrid expert as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an industrial large model anomaly detection method integrating image mask and hybrid expert as described in any one of claims 1 to 8 is implemented.

Images