Scene anomaly detection method based on multi-mode CLIP visual language model

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a scene anomaly detection method based on a multi-modal CLIP visual language model, which comprises a multi-expert adapter with multi-level features, image text feature comparison and CLIP LoRA efficient parameter fine adjustment. A common method of applying the CLIP to a downstream task is to introduce an extra cross-layer CLIP-Adapter into a VIT backbone network, that is to say, a trainable BottleNeck is introduced between different layers, and a multi-expert network of an expert hybrid model is introduced into an up-sampling part of the BottleNeck; the MoE multi-expert network is introduced into the up-sampling part of BottleNeck, modeling is carried out on different scenes and scene anomalies, each expert specially studies a subset of an input space, optimization can be carried out for specific problems, and the efficiency of processing the specific problems is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically is a scene anomaly detection method based on a multimodal CLIP visual language model. Background Art

[0002] With the rapid development of industrialization and urbanization, safety management in the factory environment and public safety fields has become increasingly important. Especially in the factory environment, any abnormal events, such as equipment failure or operational errors, may lead to serious production losses or even casualties. Similarly, in fire passages, illegal occupation may hinder rescue work in an emergency and cause immeasurable losses.

[0003] Traditional anomaly detection methods usually rely on manual features and thresholding, which have limited generalization capabilities in complex scenarios. In recent years, the development of deep learning technology has provided new possibilities for anomaly detection. In particular, methods based on convolutional neural networks have shown excellent performance in image and video analysis.

[0004] However, existing deep learning methods still face challenges when dealing with specific scenarios, such as factory environments and fire escape monitoring. First, data acquisition for these scenarios may be limited, resulting in insufficient training data. Second, the complexity and dynamics of the scenarios require the model to have higher generalization capabilities and sensitivity to anomalies.

[0005] Therefore, to address the above problems, a scene anomaly detection method based on the multimodal CLIP visual language model is proposed. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides a scene anomaly detection method based on a multimodal CLIP visual language model, which is used to solve the problems raised in the above background technology.

[0008] (II) Technical solution

[0009] To achieve the above object, the present invention provides the following technical solution: a scene anomaly detection method based on a multimodal CLIP visual language model, comprising a multi-expert adapter of multi-level features, image-text feature comparison and CLIP LoRA efficient parameter fine-tuning.

[0010] Preferably, a multi-expert adapter with multi-level features: A common practice for applying CLIP to downstream tasks is to introduce additional cross-layer CLIP-Adapters into the VIT backbone network, that is, to introduce trainable BottleNecks between different layers and introduce a multi-expert network of expert mixture models into the sampling part of BottleNeck.

[0011] Preferably, the VIT backbone network of CLIP is divided into four stages;

[0012] The first three layers introduce a CLIP-MoE-Adapter. In order to project the image features of each layer to the feature dimension and feature space consistent with the text features, the size of the intermediate dimension of BottleNeck is selected to be consistent with the text features, because the image features after the intermediate layer projection need to be compared with the text features by cosine similarity. The original input image can be represented as x∈R h×w×3 , after the Transformer layer, the image features can be represented as F vis ∈R G×d , where the height and width of the original input image are represented as h, the width is represented as w, the number of RGB channels of the image is represented as 3, the number of patches into which the image is divided before being input into VIT is represented as G, and the dimension of the embedding feature layer of the token is represented as d. According to the process from stage one to stage three, the image features can be represented as F l ∈R G×d , l∈{1, 2, 3}, there are three CLIP-MoE-Adapters in the first three stages, denoted as A l (·), l∈{1, 2, 3}, and the last stage is just a feature projection layer, i.e., the downsampling layer of BottleNeck, denoted as P(·);

[0013] The first three stages have two outputs, namely, image features F vis After the intermediate layer projection, F middle , and F vis The output after the entire CLIP-MoE-Adapter can be expressed as A l (F l ), the output of each expert network will be scored by the experts of the routing network, and the output of the expert network will be weighted and summed. The output of the routing network can be expressed as:

[0014]

[0015] Where W g is the weight matrix of the routing network, and the output of each expert can be expressed as:

[0016]

[0017] Then weighting is performed, which can be expressed as:

[0018]

[0019] Finally, this output is compared with F lMerge the output to the next layer of VIT, which can be expressed as:

[0020]

[0021] Where γ = 0.1, which is F l The weight size, It is used as the input of the next layer;

[0022] In addition, the CLIP-MoE-Adapter was applied twice at each stage, i.e., the input F l Seen as two and F l Equal input F cls，l and F seg，l , these two inputs are passed through CLIP-MoE-Adapter to obtain A cls，l (F cls，l ), A seg，l (F seg，l ), so A in the above formula l The adjustment formula can be expressed as:

[0023] A l =A cls，l +A seg，l ;

[0024] For the output of the last stage of the model, since it only passes through the projection layer P(·), it can be expressed as:

[0025]

[0026]

[0027] where F vis is the output of the last layer of the VIT backbone network, W cls , W seg They are the final stage projection weight parameters for the image-level classification task and the pixel-level classification task, respectively.

[0028] Preferably, image text feature comparison: the text prompt includes prompt words for normal image features and prompt words for abnormal images. The prompt words are generated by a set template, and features of normal prompt words and abnormal prompt words are extracted respectively. Then, the mean of the features of normal prompt words and abnormal prompt words is calculated respectively. Finally, the text features can be expressed in the feature space as F text ∈R 2×d ;

[0029] The image-level labels of normal and abnormal images in the dataset are represented as c∈{0,1}, where 0 represents normal and 1 represents abnormal. The abnormal mask of the image is represented as S i ∈{0, 1}h×w , that is, the pixel-by-pixel anomaly label of the image, where 0 represents a normal pixel and 1 represents an abnormal pixel. Based on the label, the text features of the feature space of each stage are compared and anomaly scores are generated. The loss of each stage is then obtained through the following loss function formula:

[0030]

[0031] Where l∈{1, 2, 3, 4}, Dice(·,·) is the similarity coefficient loss function, Focal(·,·) is the focal loss function, BCE(·,·) is the binary cross entropy loss function, and their respective weights are set to λ1=λ2=λ3=1.0. Finally, the loss of each stage is accumulated as the total loss The objective function used for backpropagation during training.

[0032] Preferred, CLIP LoRA efficient parameter fine-tuning: LoRA is a technology used to fine-tune large language models and diffusion models. In deep learning, large models usually require a lot of data and computing resources for training. However, in practical applications, it is often necessary to fine-tune the pre-trained large models for specific tasks. The core idea of ​​LoRA is to use low-rank matrix decomposition to approximate the changes in model parameters. During the fine-tuning process, LoRA adjusts the model by adding a low-rank matrix.

[0033] (III) Beneficial effects

[0034] Compared with the prior art, the present invention provides a scene anomaly detection method based on a multimodal CLIP visual language model, which has the following beneficial effects:

[0035] 1. This invention introduces a multi-expert network of expert mixture model in the upsampling part of BottleNeck to model different scenarios and scenario anomalies separately. Each expert specializes in a subset of the input space, which can be optimized for specific problems and improve the efficiency of processing specific problems.

[0036] 2. The invention introduces a LoRA branch to W, trains the parameters of A and B, and merges the branch into the original weights when initializing the parameters for forward reasoning, thereby achieving the effect of not introducing any additional parameter weights, greatly reducing the number of parameters that need to be updated.

[0037] 3. This invention further improves the accuracy of the model in fine-grained pixel-level performance through image-text feature comparison and CLIP LoRA efficient parameter fine-tuning.

[0038] 4. This invention, by introducing multi-level feature multi-expert Adapter into the VIT backbone network of CLIP, enables the model to learn more information and improves the richness and diversity of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 is a flow chart of the scene anomaly detection method of the present invention;

[0041] Figure 2 It is a flowchart of the prompt word feature extraction of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Specific examples are given below.

[0044] The present invention provides a scene anomaly detection method based on a multimodal CLIP visual language model, including a multi-expert adapter of multi-level features, image-text feature comparison, and CLIP LoRA efficient parameter fine-tuning;

[0045] Multi-expert adapters for multi-level features: A common approach to applying CLIP to downstream tasks is to introduce additional cross-layer CLIP-Adapters into the VIT backbone network, that is, to introduce trainable BottleNecks between different layers and introduce a multi-expert network of expert mixture models into the BottleNeck upsampling part;

[0046] By introducing the MoE multi-expert network in the upsampling part of BottleNeck, different scenarios and scenario anomalies are modeled separately. Each expert specializes in a subset of the input space, which can be optimized for specific problems and improve the efficiency of handling specific problems.

[0047] CLIP’s VIT backbone network is divided into four stages;

[0048] The first three layers introduce a CLIP-MoE-Adapter. In order to project the image features of each layer to the feature dimension and feature space consistent with the text features, the size of the intermediate dimension of BottleNeck is selected to be consistent with the text features, because the image features after the intermediate layer projection need to be compared with the text features by cosine similarity. The original input image can be represented as x∈R h×w×3 , after the Transformer layer, the image features can be represented as F vis ∈R G×d , where the height and width of the original input image are represented as h, the width is represented as w, the number of RGB channels of the image is represented as 3, the number of patches into which the image is divided before being input into VIT is represented as G, and the dimension of the embedding feature layer of the token is represented as d. According to the process from stage one to stage three, the image features can be represented as F l ∈R G×d , l∈{1, 2, 3}, there are three CLIP-MoE-Adapters in the first three stages, denoted as A l (·), l∈{1, 2, 3}, and the last stage is just a feature projection layer, i.e., the downsampling layer of BottleNeck, denoted as P(·);

[0049] The first three stages have two outputs, namely, image features F vis After the intermediate layer projection, F middle , and F vis The output after the entire CLIP-MoE-Adapter can be expressed as A l (F l ), the output of each expert network will be scored by the experts of the routing network, and the output of the expert network will be weighted and summed. The output of the routing network can be expressed as:

[0050]

[0051] Where W g is the weight matrix of the routing network, and the output of each expert can be expressed as:

[0052]

[0053] Then weighting is performed, which can be expressed as:

[0054]

[0055] Finally, this output is compared with F l Merge the output to the next layer of VIT, which can be expressed as:

[0056]

[0057] Where γ = 0.1, which is F l The weight size, It is used as the input of the next layer;

[0058] In addition, the CLIP-MoE-Adapter was applied twice at each stage, i.e., the input F l Seen as two and F l Equal input F cls，l and F seg，l , these two inputs are passed through CLIP-MoE-Adapter to obtain A cls，l (F cls，l ), A seg，l (F seg，l ), so A in the above formula l The adjustment formula can be expressed as:

[0059] A l =A cls，l +A seg，l ;

[0060] For the output of the last stage of the model, since it only passes through the projection layer P(·), it can be expressed as:

[0061]

[0062] where F vis is the output of the last layer of the VIT backbone network, W cls , W seg They are the final stage projection weight parameters for the image-level classification task and the pixel-level classification task respectively;

[0063] By introducing multi-level feature multi-expert adapters into the VIT backbone network of CLIP, the model can learn more information and improve the richness and diversity of feature extraction;

[0064] like Figure 2 As shown in the figure, image text feature comparison: the text prompt includes prompt words for normal image features and prompt words for abnormal images. The prompt words are generated through the set template, and the features of normal prompt words and abnormal prompt words are extracted respectively. Then, the mean of the features of normal prompt words and abnormal prompt words are calculated respectively. Finally, the text features can be expressed in the feature space as F text ∈R 2×d ;

[0065] The image-level labels of normal and abnormal images in the dataset are represented as c∈{0,1}, where 0 represents normal and 1 represents abnormal. The abnormal mask of the image is represented as S i ∈{0, 1} h×w, that is, the pixel-by-pixel anomaly label of the image, where 0 represents a normal pixel and 1 represents an abnormal pixel. Based on the label, the text features of the feature space of each stage are compared and anomaly scores are generated. The loss of each stage is then obtained through the following loss function formula:

[0066]

[0067] Where l∈{1, 2, 3, 4}, Dice(·,·) is the similarity coefficient loss function, Focal(·,·) is the focal loss function, BCE(·,·) is the binary cross entropy loss function, and their respective weights are set to λ1=λ2=λ3=1.0. Finally, the loss of each stage is accumulated as the total loss The objective function used for backpropagation during training.

[0068] CLIP LoRA efficient parameter fine-tuning: LoRA is a technology used to fine-tune large language models and diffusion models. In deep learning, large models usually require a lot of data and computing resources for training. However, in practical applications, it is often necessary to fine-tune pre-trained large models for specific tasks. The core idea of ​​LoRA is to use low-rank matrix decomposition to approximate the changes in model parameters. During the fine-tuning process, LoRA adjusts the model by adding a low-rank matrix. Assuming there is a pre-trained weight matrix W, two low-rank matrices A and B are introduced during the fine-tuning process, so that the new weight matrix W′ is expressed as: W′=W+ΔW, where ΔW=AB T Here, A and B are low-rank matrices, whose ranks are much smaller than the rank of W. Only A and B need to be updated instead of the entire W matrix, which greatly reduces the number of parameters that need to be updated. During training, only the LoRA branch is introduced to W, and the parameters of A and B are trained. When initializing the parameters for forward reasoning, the branch is merged into the original weights, and the effect of introducing no additional parameter weights can be achieved.

[0069] By introducing the LoRA branch to W, training the parameters of A and B, and merging the branch into the original weights when initializing the parameters for forward reasoning, the effect of not introducing any additional parameter weights can be achieved, greatly reducing the number of parameters that need to be updated;

[0070] Experimental setup

[0071] Dataset: Two typical application scenarios of scene anomaly detection were selected, namely, substation anomaly detection and fire channel occupancy. In addition, a method of artificially embedding anomalies was proposed. A certain number of normal images of substations / fire channels were selected as the background, and foreground targets covering 47 categories, such as cars, animals, tools, etc., were selected from the OPA dataset. The foreground targets were manually selected and embedded in the background to obtain anomaly maps and GT maps. Among them, 30 of the 47 categories were selected as foreground embeddings in the training set, and the remaining 17 were only used as foreground embeddings in the test set and validation set. This dataset design strategy is to fit the situations encountered in practice, that is, anomalies are often not seen during training, which can also better evaluate the zero-shot capability of the model. Finally, two datasets for substation scenarios and fire channel scenarios were designed;

[0072] The training set of the substation dataset includes the substation dataset and the fire channel dataset;

[0073] Substation data set: Training set A consists of 1487 samples. In addition, in order to evaluate the richness of the types of scenes covered by positive and negative samples in the training set and the impact of the field of view on the model performance, the training set was also subjected to sample deletion and data enhancement, namely, training set B: Based on training set A, a data enhancement strategy was adopted. By dividing the image into local images of different sizes, it was also used as part of the data set to further evaluate the performance of the CLIP model in the absence of global information and limited field of view. It consists of a total of 944 normal samples and 896 abnormal samples. Training set C: Based on training set A, samples of various substation scenes and multi-angle images of the same scene were deleted, consisting of a total of 1144 samples, while the test set and validation set consisted of 176 normal samples and 301 abnormal samples, divided in a 1:1 ratio, which also included some real substation scene anomalies that did not appear in the training set, and some samples were also subjected to data enhancement processing of image cropping for comprehensive evaluation of the performance of the model.

[0074] Fire passage data set: Training set A consists of 1084 samples. In addition, in order to evaluate the variety of scenes covered by positive and negative samples in the training set and the impact of the field of view on the model performance, the training set was also subjected to sample deletion and data enhancement, namely, training set B: Based on training set A, a data enhancement strategy was adopted. By dividing the image into local images of different sizes, it was also used as part of the data set to further evaluate the performance of the CLIP model in the absence of global information and limited field of view. It consists of a total of 944 normal samples and 896 abnormal samples. Training set C: Based on training set A, samples of various substation scenes and multi-angle images of the same scene were deleted, consisting of a total of 1144 samples, while the test set and validation set consisted of 176 normal samples and 301 abnormal samples, divided in a 1:1 ratio, which also included some real substation scene anomalies that did not appear in the training set, and some samples were also subjected to data enhancement processing of image cropping for comprehensive evaluation of the performance of the model.

[0075] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A scene anomaly detection method based on a multimodal CLIP visual language model, characterized in that: Includes multi-expert adapters with multi-level features, image-text feature comparison, and efficient parameter fine-tuning of CLIP LoRA.

2. According to claim 1, a scene anomaly detection method based on a multimodal CLIP visual language model is characterized in that: Multi-expert adapter with multi-level features: A common practice for applying CLIP to downstream tasks is to introduce additional cross-layer CLIP-Adapters into the VIT backbone network, that is, to introduce trainable BottleNeck between different layers and introduce a multi-expert network of expert mixture models into the sampling part of BottleNeck.

3. The scene anomaly detection method based on the multimodal CLIP visual language model according to claim 2, characterized in that: CLIP’s VIT backbone network is divided into four stages; The first three layers introduce a CLIP-MoE-Adapter. In order to project the image features of each layer to the feature dimension and feature space consistent with the text features, the size of the intermediate dimension of BottleNeck is selected to be consistent with the text features, and then the cosine similarity is compared with the text features. The original input image is represented as x∈R h×w×3 , after the Transformer layer, the image feature is represented as F υis ∈R G×d , where the height and width of the original input image are represented as h, the width is represented as w, the number of RGB channels of the image is represented as 3, the number of patches into which the image is divided before being input into VIT is represented as G, and the dimension of the embedding feature layer of the token is represented as d. According to the process from stage one to stage three, the image features can be represented as F l ∈R G×d , l∈{1, 2, 3}, there are three CLIP-MoE-Adapters in the first three stages, denoted as A l (·), l∈{1, 2, 3}, and the last stage is a feature projection layer, i.e., the downsampling layer of BottleNeck, denoted as P(·); The first three stages have two outputs, namely, image features F υis After the intermediate layer projection, F middle , and F υis The output after the entire CLIP-MoE-Adapter is represented by A l (F l ), the output of each expert network will be scored by the experts of the routing network, and the output of the expert network will be weighted and summed. The output of the routing network can be expressed as: Where W g is the weight matrix of the routing network, and the output of each expert can be expressed as: Then weighting is performed, which can be expressed as: Finally, this output is compared with F l Merge the output to the next layer of VIT, which can be expressed as: Where γ = 0.1, which is F l The weight size, It is used as the input of the next layer; In addition, the CLIP-MoE-Adapter was applied twice at each stage, i.e., the input F l Seen as two and F l Equal input F cls，l and F seg，l , these two inputs are each passed through the CLIP-MoE-Adapter to obtain A cls，l (F cls，l ), A seg，l (F seg，l) , so A in the above formula is l The adjustment formula can be expressed as: A l =A cls，l +A seg，l ; The output of the last stage of the model, which only passes through the projection layer P(·), can be expressed as: Among them, F υis is the output of the last layer of the VIT backbone network, W cls , W seg They are the final stage projection weight parameters for the image-level classification task and the pixel-level classification task, respectively.

4. The scene anomaly detection method based on the multimodal CLIP visual language model according to claim 1, characterized in that: Image-text feature comparison: The text prompt includes prompt words for normal image features and prompt words for abnormal images. The prompt words are generated through the set template. The features of normal prompt words and abnormal prompt words are extracted respectively, and then the mean of the features of normal prompt words and abnormal prompt words are calculated respectively. Finally, the text features can be expressed in the feature space as F text ∈R 2×d ; The image-level labels of normal and abnormal images in the dataset are represented as c∈{0,1}, where 0 represents normal and 1 represents abnormal. The abnormal mask of the image is represented as S i ∈{0, 1} h×w , that is, the pixel-by-pixel anomaly label of the image, where 0 represents a normal pixel and 1 represents an abnormal pixel. Based on the label, the text features of the feature space of each stage are compared and anomaly scores are generated. The loss of each stage is then obtained through the following loss function formula: Where l∈{1, 2, 3, 4}, Dice(·,·) is the similarity coefficient loss function, Focal(·,·) is the focal loss function, BCE(·,·) is the binary cross entropy loss function, and their respective weights are set to λ1=λ2=λ3=1.

0. Finally, the loss of each stage is accumulated as the total loss The objective function used for backpropagation during training.

5. The scene anomaly detection method based on the multimodal CLIP visual language model according to claim 1, characterized in that: CLIP LoRA efficient parameter fine-tuning: LoRA is a technology used to fine-tune large language models and diffusion models. In deep learning, pre-trained large models need to be fine-tuned for specific tasks. The core idea of ​​LoRA is to use low-rank matrix decomposition to approximate the changes in model parameters. During the fine-tuning process, LoRA adjusts the model by adding a low-rank matrix.

Images