Image anomaly detection method and system based on knowledge transfer, terminal and medium

Abstract

The application discloses a kind of based on knowledge transfer's image anomaly detection method and system, pre-training teacher network and student network are used, and initial model of image anomaly detection is constructed;Normal sample is obtained, and the knowledge of pre-training teacher network is transferred to student network by intra-frame distillation;The nearest abnormal sample between pre-training teacher network and student network is stored as inter-frame prototype feature library, the greatest difference in representation is represented;According to probability strategy, intra-frame distillation and inter-frame distillation are selected, the representation ability of normal sample and the representation ability of near abnormal sample of student network are trained respectively, and image anomaly detection model is obtained, the multi-scale feature difference of the image sample to be detected is obtained as image anomaly score, and image abnormal sample is detected.The application is conducive to model learning richer normal sample features, forms a more actual demand decision boundary, thereby reducing the false detection rate of anomaly detection, and is more suitable for application in the scene such as industrial surface defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to an anomaly detection technology in the technical field of image processing, in particular to an image anomaly detection method and system based on knowledge transfer, a terminal and a medium. BACKGROUND

[0002] Anomaly detection is a hot research direction in the field of computer vision, and has high commercial value and application value in public security, financial supervision, network security, e-commerce transactions and other fields. Product surface anomaly detection is also one of the important application fields of anomaly detection. Most anomaly detection methods only use normal samples to train the model, so that the model can detect abnormal samples deviating from the normal paradigm, which belongs to unsupervised learning.

[0003] In recent years, many anomaly detection methods use reconstruction mechanism to construct anomaly detection model (Improving unsupervised defect segmentation by applying structural similarity to autoencoders, 2018). These methods only train the model to reconstruct normal samples, so the model has relatively poor reconstruction ability for abnormal samples, so as to realize anomaly detection. Another type of anomaly detection method is based on knowledge distillation to realize anomaly detection. These methods use a pre-trained teacher network to train a student network to learn the teacher network's representation of normal samples. Therefore, when the input is an abnormal sample, the student network's representation of the abnormal sample will be very different from the teacher network's representation, thereby realizing anomaly detection.

[0004] However, in some actual industrial scenarios, the above methods are very sensitive to any abnormal disturbance on the surface of the product, resulting in model false detection. For example, some normal samples of the KSDD2 dataset have some irregularly distributed small white spots; some normal samples of the Magnetic tile dataset have uneven brightness; some normal samples of the AITEX fabric dataset have a small amount of long stripes. These disturbances or non-defect "visual defects" appearing on normal samples are easy to be misdetected by anomaly detection models as abnormal, and we call these misdetected samples as near-abnormal samples. The main reason for the misdetection of near-abnormal samples is that most anomaly detection methods are based on CNN architecture, only focusing on the local receptive field of the sample, and unable to establish global dependency, making it difficult for the model to learn near-abnormal sample features well. While some anomaly detection methods based on Transformer architecture use attention mechanisms, model training is also limited to intra-frame features of a single image.

[0005] Therefore, how to improve the representation ability of the abnormal detection model to normal samples, thereby reducing the false detection rate of normal samples, and without increasing the missed detection rate of abnormal samples, has extremely high research value and practical significance.

[0006] Currently, no similar technology to the present application has been found to be described or reported, and no similar data has been collected at home and abroad. SUMMARY

[0007] The present application aims at the above-mentioned deficiencies in the prior art, and proposes an image anomaly detection method, system, terminal and medium based on knowledge transfer using near abnormal samples.

[0008] According to one aspect of the present application, an image anomaly detection method based on knowledge transfer is provided, comprising:

[0009] An initial image anomaly detection model is constructed using a pre-trained teacher network and a student network.

[0010] Normal images are obtained as normal samples, and the knowledge of the normal samples in the pre-trained teacher network is transferred to the student network through intra-frame distillation.

[0011] The abnormal images with the greatest representation difference between the pre-trained teacher network and the student network are stored as near abnormal samples, forming an inter-frame prototype feature library.

[0012] The normal samples and the inter-frame prototype feature library are used to select intra-frame distillation and inter-frame distillation according to a probability strategy, and the representation ability of the normal samples and the representation ability of the near abnormal samples of the student network are trained respectively, to obtain an image anomaly detection model.

[0013] The image anomaly detection model is used to obtain the multi-scale feature difference of the image sample to be detected as an image anomaly score, and the image abnormal sample is detected.

[0014] Preferably, the initial image anomaly detection model is constructed using a pre-trained teacher network and a student network, comprising:

[0015] A pre-trained Transformer model on a data set is selected as a pre-trained teacher network.

[0016] A student network is constructed, and the architecture of the student network is adapted to the architecture of the pre-trained teacher network.

[0017] The pre-trained teacher network and the student network are used as the main architecture to construct an initial image anomaly detection model.

[0018] Preferably, the normal image is obtained as a normal sample, and knowledge of the normal sample in the pre-trained teacher network is migrated to the student network through intra-frame distillation, comprising:

[0019] The normal image is obtained as a normal sample, and the representation ability of the normal sample in the pre-trained teacher network is migrated to the student network through multi-scale knowledge distillation within the frame by using the normal sample.

[0020] Preferably, the abnormal image with the largest representation difference between the pre-trained teacher network and the student network is stored as a near abnormal sample to form an inter-frame prototype feature library, comprising:

[0021] The normal sample is simultaneously input into the pre-trained teacher network and the student network of the image anomaly detection initial model, and the representation of the normal sample in multiple different scales of the two networks is obtained respectively;

[0022] The similarity of the multi-scale features of the normal sample in the two networks is calculated by similarity measurement;

[0023] According to the similarity of the multi-scale features, N abnormal samples with the lowest similarity, i.e. the largest representation difference, are obtained as near abnormal samples;

[0024] The multi-scale features of the N near abnormal samples are extracted by using the pre-trained teacher network;

[0025] The extracted multi-scale features of the near abnormal samples are stored as inter-frame prototype features to form an inter-frame prototype feature library.

[0026] Preferably, the intra-frame distillation and inter-frame distillation are selected according to the probability strategy, and the representation ability of the normal sample and the representation ability of the near abnormal sample of the student network are trained respectively, comprising:

[0027] An hyperparameter λ is defined as an inter-frame distillation probability, and the value of λ is 0-1;

[0028] In each round of training, inter-frame distillation is performed with a probability of λ, and the representation ability of the student network for the near abnormal sample is optimized through the inter-frame distillation;

[0029] In each round of training, intra-frame distillation is performed with a probability of 1-λ, and the representation ability of the student network for the normal sample is optimized through the intra-frame distillation.

[0030] Preferably, the intra-frame distillation with a probability of 1-λ comprises:

[0031] The normal samples are taken as inputs of the teacher network and the student network to perform intra-frame distillation, wherein the intra-frame distillation comprises: simultaneously inputting each normal sample into the teacher network and the student network, and each of the two networks obtains a multi-scale representation of the normal sample;

[0032] The student network learns the representation of the normal sample by the teacher network through the similarity loss, and optimizes the representation ability of the student network for the normal sample.

[0033] Preferably, the inter-frame distillation with a probability of λ comprises:

[0034] The normal samples are taken as inputs of the teacher network and the student network to perform inter-frame distillation by introducing inter-frame prototype features into the intermediate layers of the two networks through an inter-frame cross attention model, and the representation ability of the student network for the near abnormal sample is optimized.

[0035] Preferably, the inter-frame cross attention model comprises: a spatial attention calculation unit, a channel response weight calculation unit and an inter-frame cross attention feature fusion unit.

[0036] The spatial attention calculation unit performs matrix multiplication on the intermediate layer output feature of the teacher network or the intermediate layer output feature of the student network and the most similar prototype feature, then compresses the channel to one dimension by taking the mean value, and then obtains the spatial attention SA through softmax as follows:

[0037]

[0038] wherein f i T is the output feature of the i-th intermediate layer of the teacher network, f i S is the output feature of the i-th intermediate layer of the student network, p i is the most similar prototype feature to f i , S(·) is a softamx operation, M(·) is a mean value operation in the channel dimension, f i represents f i T or f i S ;

[0039] The channel response weight calculation unit performs average pooling on p i , and then performs softmax to obtain the channel response weight CW as follows:

[0040] CW=S((A(p i ))

[0041] wherein A(·) is an average pooling operation;

[0042] The inter-frame cross attention feature fusion unit performs a dot multiplication operation on the spatial attention SA and the channel response weight CW to obtain the inter-frame cross attention; the inter-frame cross attention is combined with f i T or f i S Performing dot multiplication operation, we get the inter-frame cross attention feature ICF:

[0043] ICF=(SA⊙CW)⊙f i

[0044] Among them, ⊙ is the dot multiplication operation.

[0045] Preferably, the method of introducing inter-frame prototype features in the intermediate layers of the two networks using the inter-frame cross attention model includes:

[0046] Obtaining the intermediate layer output features of the teacher network and the intermediate layer output features of the student network;

[0047] Selecting the most similar prototype feature from the inter-frame prototype feature library according to the intermediate layer output feature of the teacher network;

[0048] Using the intermediate layer output features of the teacher network and the most similar prototype features as inputs to the inter-frame cross attention model, obtaining inter-frame cross attention features, and using them as inputs to the next layer of the teacher network;

[0049] The intermediate layer output features of the student network and the most similar prototype features are used as inputs of the inter-frame cross-attention model to obtain inter-frame cross-attention features, which are used as inputs of the next layer of the student network.

[0050] Preferably, the inter-frame distillation includes: enabling the student network to learn the inter-frame cross-attention features between the input normal samples and the inter-frame prototype features of the teacher network.

[0051] Preferably, the method of using the image anomaly detection model to obtain multi-scale feature differences of the image samples to be detected as image anomaly scores and detecting abnormal image samples includes:

[0052] Obtain an image sample to be detected;

[0053] Passing the image sample to be detected through the teacher network and the student network of the image anomaly detection model respectively to obtain multi-scale features;

[0054] Calculating a difference map of the multi-scale features using a similarity metric;

[0055] interpolating and fitting the difference map of the multi-scale features, and adding values of corresponding positions of the difference map of the multi-scale features to obtain an image anomaly score map of the same scale as the image sample to be detected;

[0056] If the image anomaly score map is less than a set threshold, it is determined as a normal image, otherwise, it is determined as an abnormal image.

[0057] According to another aspect of the present application, there is provided an image anomaly detection system based on knowledge transfer, comprising:

[0058] an anomaly detection model module, which adopts a pre-trained teacher network and a student network, constructs an initial anomaly detection model, and uses a final anomaly detection model to obtain a multi-scale feature difference of an image sample to be detected as an image anomaly score to detect an image anomaly sample;

[0059] a knowledge transfer module, which uses a normal image obtained as a normal sample, and transfers normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation;

[0060] an inter-frame prototype feature library, which is used to store an abnormal image with the greatest representation difference between the pre-trained teacher network and the student network as a near abnormal sample to form an inter-frame prototype feature library;

[0061] a student network training module, which uses the normal sample and the inter-frame prototype feature library, selects intra-frame distillation and inter-frame distillation according to a probability strategy, respectively trains normal sample representation ability and near abnormal sample representation ability of the student network, and obtains a final anomaly detection model.

[0062] According to a third aspect of the present application, there is provided a computer terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to execute the method of any one of the above, or runs the system of the above.

[0063] According to a fourth aspect of the present application, there is provided a computer readable storage medium, which stores a computer program executable by a processor to execute the method of any one of the above, or run the system of the above.

[0064] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0065] The present application uses near abnormal samples to realize an image anomaly detection method based on knowledge transfer, which is conducive to model learning of more abundant normal sample features, forms a decision boundary more in line with actual needs, and thus reduces the false detection rate of anomaly detection.

[0066] The application adopts an inter-frame distillation method, can introduce near abnormal sample features by using an inter-frame cross attention model, and effectively alleviates the false detection of near abnormal samples.

[0067] The application adopts a multi-scale knowledge distillation technology for modeling and abnormal score estimation, and can realize real-time abnormal detection. BRIEF DESCRIPTION OF DRAWINGS

[0068] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0069] Figure 1 The figure is a workflow diagram of the image abnormality detection method based on knowledge transfer in a preferred embodiment of the application.

[0070] Figure 2 The figure is a training process diagram of the abnormality detection model in a preferred embodiment of the application; (a) is intra-frame distillation, and (b) is inter-frame distillation.

[0071] Figure 3 The figure is a structure and working process diagram of the inter-frame cross attention model in a preferred embodiment of the application.

[0072] Figure 4 The figure is a detection workflow diagram of the abnormality detection model in a preferred embodiment of the application.

[0073] Figure 5 The figure is a composition module diagram of the image abnormality detection system based on knowledge transfer in a preferred embodiment of the application. DETAILED DESCRIPTION

[0074] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These all belong to the protection scope of the application.

[0075] An embodiment of the application provides an image abnormality detection method based on knowledge transfer, which uses near abnormal samples for knowledge transfer optimization to form a decision boundary more in line with actual needs, thereby reducing the false detection rate of abnormality detection and being more suitable for application in industrial vision and other scenes.

[0076] As shown in (a) and (b) of Figure 1 and Figure 2 The image abnormality detection method based on knowledge transfer provided by this embodiment can include:

[0077] S1, uses the pre-trained teacher network and student network to build the initial model for image anomaly detection;

[0078] S2, obtains normal images as normal samples, and transfers the normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation;

[0079] S3, stores the abnormal images with the largest representation difference between the pre-trained teacher network and the student network as near-abnormal samples to form an inter-frame prototype feature library;

[0080] S4, using the normal sample and inter-frame prototype feature library, selects intra-frame distillation and inter-frame distillation according to the probability strategy, and trains the student network's normal sample representation ability and near-abnormal sample representation ability respectively, to obtain the image anomaly detection model;

[0081] S5, using the image anomaly detection model, obtains the multi-scale feature differences of the image samples to be detected as the image anomaly scores, and detects the image anomaly samples.

[0082] In a preferred embodiment of S1, a pre-trained teacher network and student network are used to construct an anomaly detection model, which may include:

[0083] S11, select a Transformer model pre-trained on the dataset as the pre-trained teacher network;

[0084] S12, build a student network whose architecture is compatible with the architecture of the pre-trained teacher network;

[0085] S13, using the pre-trained teacher network and student network as the main architecture, constructs the initial model for image anomaly detection.

[0086] In a preferred embodiment of S2, obtaining a normal image as a normal sample and transferring the normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation may include:

[0087] S21, obtain normal images as normal samples, and use the normal samples to transfer the representation ability of normal samples in the pre-trained teacher network to the student network through multi-scale knowledge distillation within the frame.

[0088] In an optional embodiment of S1 and S2, intra-frame distillation is used to transfer knowledge from a pre-trained teacher network to a student network. A SwinTransformer-B model pre-trained on ImageNet can be selected as the teacher network. Of course, in other embodiments, other models such as the DeiT model and the ViT model can also be used, and are not limited to the SwinTransformer-B model.

[0089] In an optional embodiment of S1 and S2, the anomaly detection model of the T-S architecture is constructed for intra-frame distillation, which can refer to the following operations:

[0090] S11, selecting a SwinTransformer-B model pre-trained on ImageNet as the teacher network;

[0091] S12, designing a student network similar in architecture to the teacher network in S11;

[0092] S21, using normal samples to transfer the knowledge of the teacher network to the student network through multi-scale knowledge distillation.

[0093] In this step, a student network similar in architecture to the teacher network is used, and multi-scale knowledge distillation is performed using normal samples. The patch-wise attention mechanism of SwinTransformer can be used to learn intra-frame and inter-frame correlations. Similar T-S architecture is easy to train, allowing the student network to learn normal sample features more efficiently.

[0094] In a preferred embodiment of S3, the most representative difference between the pre-trained teacher network and the student network is stored as a near-anomaly sample to form an inter-frame prototype feature library, which can include:

[0095] S31, inputting normal samples into the pre-trained teacher network and student network of the image anomaly detection initial model to obtain multiple different scale representations of the normal samples by the two networks respectively;

[0096] S32, calculating the similarity of the multi-scale features of the normal samples by the two networks through similarity measurement;

[0097] S33, obtaining N abnormal images with the lowest similarity, i.e., the most representative difference, as near-anomaly samples according to the similarity of the multi-scale features;

[0098] S34, extracting multi-scale features of the N near-anomaly samples using the pre-trained teacher network;

[0099] S35, storing the extracted multi-scale features of the near-anomaly samples as inter-frame prototype features to form an inter-frame prototype feature library.

[0100] In an optional example of S3, the near-anomaly samples with the most representative difference between the teacher and student networks can be directly stored as an inter-frame prototype feature library. Of course, in other embodiments, trainable memory libraries, Codebook, and other prototype storage methods can also be used, and are not limited to direct storage.

[0101] In an optional embodiment, abnormal images with the largest representation difference between the pre-trained teacher network and the student network are stored as near-abnormal samples to form an inter-frame prototype feature library. The following operations can be referred to:

[0102] S31, input the normal samples into the pre-trained teacher network and student network of the initial model of image anomaly detection at the same time, and obtain the representations of the normal samples at multiple different scales by the two networks respectively;

[0103] S32, through the cosine similarity measurement, obtain the multi-scale feature similarity graph of the teacher and student networks for normal samples

[0104]

[0105] in, is the output feature map of the i-th intermediate layer of the teacher network, is the output feature map of the i-th intermediate layer of the student network, C i 、H i and W i are the number of channels, height, and width of the output feature map, respectively.

[0106] S33, obtain the N near-abnormal samples with the lowest similarity according to the feature similarity graph of S32;

[0107] S34, using the pre-trained teacher network to extract the multi-scale features of the near-abnormal samples in S33, that is, to obtain the near-abnormal features;

[0108] S35, storing the extracted near-abnormal features as an inter-frame prototype feature library P.

[0109] In this step, multi-scale features refer to the image input network, and the network will output feature maps of multiple scales (for convenience of explanation, f i to represent the feature map of the i-th scale, i = 0, 1, 2); Similarity metric: refers to the f of the teacher network i F with student network i Similarity is calculated between them; Features: A neural network is a cascade of multiple modules. After the image is input into the neural network, these multiple modules will have an output, which is the feature; Scale: The size (length and width) of the image or feature. The later the module in the neural network, the smaller the scale obtained. For example, if the image is 384x384 at the beginning, after being input into the neural network, the output features of multiple scales obtained are 48x48, 24x24, 12x12, ... in order.

[0110] In this step, a similarity metric is used to identify N normal samples that are most difficult for the student network to learn, namely, near-abnormal samples. Features of these near-abnormal samples are then extracted using a pre-trained teacher network and stored as an inter-frame prototype library. Using a similarity metric to find near-abnormal samples is simple and effective, and using the teacher network to extract and store near-abnormal features is straightforward and convenient.

[0111] In a preferred embodiment of S4, intra-frame distillation and inter-frame distillation are selected according to a probabilistic strategy to train the student network's ability to represent normal samples and nearly abnormal samples, respectively, which may include:

[0112] S41, define a hyperparameter λ as the inter-frame distillation probability, and the value of λ ranges from 0 to 1;

[0113] S42, in each round of training, inter-frame distillation is performed with a probability of λ to optimize the student network's ability to represent near-abnormal samples through inter-frame distillation;

[0114] S43, in each round of training, intra-frame distillation is performed with a probability of 1-λ to optimize the student network's representation ability for normal samples through intra-frame distillation.

[0115] In a preferred embodiment of S43, performing intra-frame distillation with a probability of 1-λ may include:

[0116] S431, using the normal sample as the input of the teacher network and the student network, and performing intra-frame distillation. The intra-frame distillation includes: inputting each normal sample into the teacher network and the student network simultaneously, and the two networks each obtain a multi-scale representation of the normal sample;

[0117] S432, the student network learns the teacher network’s representation of normal samples through similarity loss, and optimizes the student network’s representation ability of normal samples.

[0118] In a preferred embodiment of S42, performing inter-frame distillation with a probability of λ may include:

[0119] S421 uses normal samples as input to the teacher network and the student network, and uses the inter-frame cross-attention model to introduce inter-frame prototype features in the middle layer of each of the two networks to perform inter-frame distillation and optimize the student network's ability to represent near-abnormal samples.

[0120] In a preferred embodiment of S421, the inter-frame cross attention model includes: a spatial attention calculation unit, a channel response weight calculation unit, and an inter-frame cross attention feature fusion unit; the spatial attention and channel response weights between images are calculated by the spatial attention calculation unit and the channel response weight calculation unit, respectively, and then the inter-frame cross attention features are obtained by fusing them by the inter-frame cross attention feature fusion unit; wherein:

[0121] The spatial attention calculation unit performs matrix multiplication on the intermediate layer output feature of the teacher network or the intermediate layer output feature of the student network and the most similar prototype feature, then compresses the channel to 1 dimension by taking the mean value, and then obtains the spatial attention SA by using softmax:

[0122]

[0123] wherein f i T is the output feature of the i-th intermediate layer of the teacher network, f i s is the output feature of the i-th intermediate layer of the student network, p i is the most similar prototype feature to f i , S(·) is a softamx operation, M(·) is a mean value taking operation in the channel dimension, f i represents f i T or f i S ;

[0124] The channel response weight calculation unit performs average pooling on p i , and then performs softmax to obtain the channel response weight CW:

[0125] CW=S((A(p i ))

[0126] wherein S(·) is a softamx operation, and A(·) is an average pooling operation;

[0127] The inter-frame cross-attention feature fusion unit performs point multiplication operation on the spatial attention SA and the channel response weight CW to obtain the inter-frame cross-attention; the inter-frame cross-attention is multiplied with f i T or f i S to obtain the inter-frame cross-attention feature ICF:

[0128] ICF=(SA⊙CW)⊙f i

[0129] wherein is a point multiplication operation.

[0130] Although the environment change will cause the teacher network to have differences in representing normal samples, the student network can still obtain normal features similar to the teacher network by introducing the inter-frame feature, so the model has the environment self-adaptation ability; the model structure and working process can be as shown in Figure 3 ;

[0131] In a preferred embodiment of S421, the inter-frame distillation includes: enabling the student network to learn the inter-frame cross-attention features between the input normal samples and the inter-frame prototype features of the teacher network.

[0132] In a preferred embodiment of S421, using an inter-frame cross attention model to introduce inter-frame prototype features in the middle layer of the student network may include:

[0133] S4211, obtaining the intermediate layer output features of the teacher network and the intermediate layer output features of the student network;

[0134] S4212, selecting the most similar prototype feature from the inter-frame prototype feature library based on the intermediate layer output features of the teacher network;

[0135] S4213, using the intermediate layer output features of the teacher network and the most similar prototype features as the input of the inter-frame cross attention model, obtains the inter-frame cross attention features, and uses them as the input of the next layer of the teacher network;

[0136] S4214, the intermediate layer output features of the student network and the most similar prototype features are used as the input of the inter-frame cross attention model to obtain the inter-frame cross attention features, which are used as the input of the next layer of the student network.

[0137] In an optional embodiment of S4, intra-frame distillation optimizes the student network's representation of normal samples, while inter-frame distillation utilizes an inter-frame cross-attention model to improve the student network's representation of near-abnormal samples. The inter-frame cross-attention model uses a non-parametric design, incorporating inter-frame prototype features into the distillation paradigm. In this step, the teacher network is fixed, and the optimization in this step is performed on the student network. Intra-frame distillation allows the student network to learn the teacher network's ability to represent a normal sample, while inter-frame distillation allows the student network to learn the teacher network's ability to represent inter-image correlations.

[0138] In this step, by using near-abnormal samples for knowledge transfer optimization, the model can learn richer normal sample features and form a decision boundary that better meets actual needs, thereby reducing the false detection rate of anomaly detection and making it more suitable for applications in scenarios such as industrial vision.

[0139] Furthermore, the implementation of the inter-frame cross attention model can refer to the following operations:

[0140] Using normal samples as the input of the network, the inter-frame cross attention module is used to introduce inter-frame prototype features in the middle layer of the network to achieve inter-frame distillation and improve the student network's ability to represent near-abnormal samples. Figure 3 shown. Figure 3In the image, both the image and the feature map are three-dimensional, with height, width, and number of channels. H: height, W: width, C: number of channels. RGB images have three channels: R, G, and B. The number of channels of the feature maps of different scales of the neural network is related to the number of convolution kernels, which are generally 1024, 512, or 256. Mean: mean, Softmax: normalized exponential function, Transpose: matrix transposition, Matrix mult: matrix multiplication, and Element-wise mult: matrix dot multiplication. In this inter-frame cross-attention model, the feature f is output by the middle layer of the network. i and prototype feature p as the input of the inter-frame cross attention model, first perform the maximum pooling operation on the prototype feature p to obtain the channel response weight, and then i Finally, the channel response weight and the spatial attention are multiplied to obtain the inter-frame cross attention feature.

[0141] In this step, the non-parametric inter-frame cross-attention model can not only effectively introduce the features of near-abnormal samples, but also avoid module overfitting caused by insufficient supervision information.

[0142] In a preferred embodiment, the inter-frame cross attention module introduces inter-frame prototype features into the distillation paradigm. Of course, in other embodiments, inter-frame prototype features can be introduced into other model paradigms and are not limited to the distillation paradigm.

[0143] Furthermore, the inter-frame cross-attention model introduces inter-frame prototype features into the distillation paradigm, which can be done as follows:

[0144] S4211, obtain the intermediate layer output feature f of the teacher network i T And the intermediate layer output feature f of the student network i S ;

[0145] S4212, based on the intermediate layer output feature f of the teacher network i T Select the most similar prototype p from the inter-frame prototype library P;

[0146] S4213, with the intermediate layer output feature f of the teacher network i T The most similar prototype p is used as the input of the inter-frame cross-attention module to obtain the inter-frame cross-attention features and serve as the input of the next layer of the teacher network.

[0147] S4214, same as S4213, uses the intermediate layer output feature f of the student network iS and the most similar prototype p as the input of the inter-frame cross attention module, to obtain the inter-frame cross attention feature, and as the input of the next layer of the student network.

[0148] In this step, the output feature of the teacher network is used to select the best prototype, and then the prototype and the output feature of the teacher (student) network are calculated through the inter-frame cross attention model to obtain the inter-frame cross feature as the input of the next level network of the teacher (student) network, so as to introduce the prototype feature into the distillation paradigm. This method of indirectly introducing the prototype avoids introducing redundant features in the prototype into the model, reducing the possibility of overfitting.

[0149] In a preferred example, the cosine similarity is used to calculate the difference between the multi-scale features of the sample by the teacher network and the student network as the anomaly score. Of course, in other embodiments, other distance measurement methods such as Euclidean distance, SIMM, etc. can be used, and it is not limited to the cosine similarity measurement method.

[0150] In a preferred embodiment of S5, the multi-scale feature difference of the image sample to be detected is obtained by using the image anomaly detection model as the image anomaly score, and the image anomaly sample is detected, which can include:

[0151] S51, obtaining an image sample to be detected;

[0152] S52, passing the image sample to be detected through the teacher network and the student network of the image anomaly detection model respectively to obtain multi-scale features;

[0153] S53, calculating the difference map of the multi-scale features by using the similarity measurement;

[0154] S54, interpolating and fitting the difference map of the multi-scale features, and adding the values of the corresponding positions of the difference map of the multi-scale features to obtain an image anomaly score map of the same scale as the image sample to be detected;

[0155] S55, if the image anomaly score map is less than a set threshold, it is determined as a normal image; otherwise, it is determined as an abnormal image.

[0156] In an optional embodiment of S5, as shown in Figure 4 the difference between the multi-scale features of the sample by the teacher network and the student network as the anomaly score can refer to the following operations:

[0157] S51, obtaining a test image sample;

[0158] S52, obtaining multi-scale features by passing the test image sample through the teacher network and the student network of the image anomaly detection model respectively;

[0159] S53, obtain a multi-scale feature difference map by using cosine similarity calculation;

[0160] S54, interpolate and fit the feature difference maps of different scales, interpolate and fit the scales of the feature difference maps of different scales to be consistent, then add the values of the corresponding positions of the multi-scale feature difference maps to obtain an image anomaly score map of the same scale as the test image sample;

[0161] S55, if the image anomaly score map is less than a set threshold, it is determined to be a normal sample; if the anomaly score value is greater than the set threshold, it is determined to be an abnormal sample.

[0162] In this step, a plurality of different scale representation difference maps are obtained by using the similarity of the feature maps, and then an anomaly score map of the same scale as the test sample is obtained by difference fitting. The method is simple, and the two-dimensional characteristics of the anomaly score map can be used to realize anomaly positioning.

[0163] The technical solutions provided by the above embodiments of the application will be further described in conjunction with a specific application example.

[0164] In this specific application example, the industrial cloth is taken as a specific application of the image anomaly detection method based on knowledge transfer, and the process of the industrial cloth anomaly detection includes the following steps.

[0165] S91, an industrial camera is used to collect a cloth image without anomaly as a normal sample.

[0166] S92, the knowledge of the normal sample of S91 in the pre-trained teacher network is transferred to the student network through intra-frame distillation;

[0167] S93, the nearest abnormal sample with the largest representation difference between the pre-trained teacher network and the student network is stored as an inter-frame prototype feature library;

[0168] S94, the normal sample and the inter-frame prototype feature library are used to select intra-frame distillation and inter-frame distillation according to a probability strategy, and the representation ability of the normal sample and the representation ability of the nearest abnormal sample of the student network are trained respectively to obtain a trained cloth surface anomaly detection model;

[0169] S95, the trained cloth surface anomaly detection model is used for deployment and application in an actual scene; the industrial camera collects the surface image of the cloth sample to be detected in real time, the image anomaly detection model obtains multi-scale feature difference, interpolates and fits and sums to obtain an anomaly score, and the position with a higher anomaly score is more likely to be abnormal.

[0170] An embodiment of the application provides an image anomaly detection system based on knowledge transfer.

[0171] As Figure 5As shown, the image anomaly detection system based on knowledge transfer provided by this embodiment may include:

[0172] Anomaly detection model module: This module uses pre-trained teacher and student networks to build an initial anomaly detection model. It then uses the final anomaly detection model to obtain multi-scale feature differences of the image samples to be detected, which are used as image anomaly scores to detect abnormal image samples.

[0173] The knowledge transfer module uses the acquired normal images as normal samples and transfers the normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation;

[0174] Inter-frame prototype feature library: This module is used to store abnormal images with the largest representation difference between the pre-trained teacher network and the student network as near-abnormal samples to form an inter-frame prototype feature library;

[0175] The student network training module uses the normal sample and inter-frame prototype feature library, selects intra-frame distillation and inter-frame distillation according to the probability strategy, and trains the student network's normal sample representation ability and near-abnormal sample representation ability respectively to obtain the final anomaly detection model.

[0176] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be elaborated here.

[0177] An embodiment of the present invention provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the processor can be used to execute any one of the methods described in the foregoing embodiments of the present invention, or to execute any one of the systems described in the foregoing embodiments of the present invention.

[0178] Optionally, the memory is configured to store a computer program. The memory can include volatile memory (for example, random access memory, abbreviated as RAM, such as static random access memory, abbreviated as SRAM, double data rate synchronous dynamic random access memory, abbreviated as DDR SDRAM, etc.), and can also include non-volatile memory (for example, flash memory). The memory is configured to store computer programs (for example, application programs, functional modules, etc. for implementing the above method), computer instructions, etc. The above computer programs, computer instructions, etc. can be stored in one or more memories in a partitioned manner. Moreover, the above computer programs, computer instructions, data, etc. can be invoked by the processor.

[0179] The above computer programs, computer instructions, etc. can be stored in one or more memories in a partitioned manner. Moreover, the above computer programs, computer instructions, data, etc. can be invoked by the processor.

[0180] The processor is configured to execute the computer program stored in the memory, so as to implement each step in the method or each module of the system according to the above embodiments. For details, refer to the related description in the above method and system embodiments.

[0181] The processor and the memory can be independent structures, or can be integrated into an integrated structure. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0182] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The program is configured to be executed by a processor, so as to implement the method in any one of the above embodiments of the present application, or run the system in any one of the above embodiments of the present application.

[0183] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0184] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks.

[0185] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks.

[0186] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks.

[0187] The knowledge transfer-based image anomaly detection method, system, terminal and medium provided by the above-mentioned embodiments of the present application use near-anomaly samples to perform knowledge transfer optimization, which is conducive to model learning of more abundant normal sample features, forms a decision boundary more in line with actual needs, thereby reducing the false detection rate of anomaly detection, and is more suitable for application in industrial vision and other scenes.

[0188] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0189] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for image anomaly detection based on knowledge transfer, characterized in that: include: Use pre-trained teacher network and student network to build the initial model for image anomaly detection; Acquire a normal image as a normal sample, and transfer the normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation; Storing abnormal images with the largest representation difference between the pre-trained teacher network and the student network as near-abnormal samples to form an inter-frame prototype feature library; Using the normal samples and the inter-frame prototype feature library, intra-frame distillation and inter-frame distillation are selected according to a probabilistic strategy to train the normal sample representation ability and the near-abnormal sample representation ability of the student network respectively, thereby obtaining an image anomaly detection model; The image anomaly detection model is used to obtain multi-scale feature differences of the image samples to be detected, which are used as image anomaly scores to detect abnormal image samples.

2. The image anomaly detection method based on knowledge transfer according to claim 1, characterized in that: The pre-trained teacher network and student network are used to build an initial model for image anomaly detection, including: Select a Transformer model pre-trained on the dataset as the pre-trained teacher network; Constructing a student network, wherein the architecture of the student network is adapted to the architecture of the pre-trained teacher network; The pre-trained teacher network and the student network are used as the main architecture to construct an initial model for image anomaly detection; The obtaining of a normal image as a normal sample and migrating normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation includes: A normal image is obtained as a normal sample, and the normal sample is used to transfer the representation ability of the normal sample in the pre-trained teacher network to the student network through multi-scale knowledge distillation within the frame.

3. The image anomaly detection method based on knowledge transfer according to claim 1, characterized in that: The storing of abnormal images with the largest representation difference between the pre-trained teacher network and the student network as near-abnormal samples to form an inter-frame prototype feature library includes: Inputting the normal samples into the pre-trained teacher network and student network of the initial model for image anomaly detection simultaneously, and obtaining representations of the normal samples at multiple different scales by the two networks respectively; Calculating the similarity of the multi-scale features of the normal samples by the two networks through similarity measurement; According to the similarity of the multi-scale features, N abnormal images with the lowest similarity, i.e., the largest representation differences, are obtained as near-abnormal samples; Extracting multi-scale features of N near-abnormal samples using the pre-trained teacher network; The extracted multi-scale features of the near-abnormal samples are stored as inter-frame prototype features to form an inter-frame prototype feature library.

4. The image anomaly detection method based on knowledge transfer according to claim 1, characterized in that: The intra-frame distillation and inter-frame distillation are selected according to the probability strategy to respectively train the normal sample representation ability and the near-abnormal sample representation ability of the student network, including: Define a hyperparameter λ as the inter-frame distillation probability, and the value of λ ranges from 0 to 1; In each round of training, inter-frame distillation is performed with a probability of λ, and the representation ability of the student network for near-abnormal samples is optimized by the inter-frame distillation; In each round of training, intra-frame distillation is performed with a probability of 1-λ, and the representation ability of the student network for normal samples is optimized through the intra-frame distillation.

5. The image anomaly detection method based on knowledge transfer according to claim 4, characterized in that: The intra-frame distillation is performed with a probability of 1-λ, including: The normal samples are used as inputs of the teacher network and the student network to perform intra-frame distillation; wherein the intra-frame distillation includes: inputting each normal sample into the teacher network and the student network simultaneously, and the two networks each obtain a multi-scale representation of the normal sample; The student network learns the representation of normal samples by the teacher network through similarity loss, thereby optimizing the representation ability of the student network for normal samples; The inter-frame distillation with a probability of λ includes: The normal samples are used as the input of the teacher network and the student network, and the inter-frame cross-attention model is used to introduce inter-frame prototype features in the middle layers of the two networks to perform inter-frame distillation to optimize the student network's ability to represent near-abnormal samples.

6. The image anomaly detection method based on knowledge transfer according to claim 5, characterized in that: The inter-frame cross attention model includes: a spatial attention calculation unit, a channel response weight calculation unit and an inter-frame cross attention feature fusion unit; wherein: The spatial attention calculation unit performs matrix multiplication on the intermediate layer output features of the teacher network or the intermediate layer output features of the student network and the most similar prototype features, then takes the mean of the channels and compresses them into 1 dimension, and then passes softmax to obtain the spatial attention SA as: Among them, f i T is the output feature of the i-th intermediate layer of the teacher network, f i S Output feature of the i-th intermediate layer of the student network, p i For and f i The most similar prototype feature, S(·) is the softamx operation, M(·) is the mean in the channel dimension, f i Represents f i T or f i S ; The channel response weight calculation unit calculates p i Perform average pooling and then softmax to obtain the channel response weight CW: CW=S((A(p i )) Where A(·) is the average pooling operation; The inter-frame cross attention feature fusion unit performs a dot product operation on the spatial attention SA and the channel response weight CW to obtain the inter-frame cross attention; the inter-frame cross attention is combined with f i T or f i S Performing dot multiplication operation, we get the inter-frame cross attention feature ICF: ICF=(SA⊙CW)⊙f i Among them, ⊙ is the dot multiplication operation; The inter-frame cross attention model is used to introduce inter-frame prototype features in the middle layers of the two networks, including: Obtaining the intermediate layer output features of the teacher network and the intermediate layer output features of the student network; Selecting the most similar prototype feature from the inter-frame prototype feature library according to the intermediate layer output feature of the teacher network; Using the intermediate layer output features of the teacher network and the most similar prototype features as inputs to the inter-frame cross attention model, obtaining inter-frame cross attention features, and using them as inputs to the next layer of the teacher network; The intermediate layer output features of the student network and the most similar prototype features are used as inputs of the inter-frame cross attention model to obtain inter-frame cross attention features, which are used as inputs of the next layer of the student network; The inter-frame distillation includes: enabling the student network to learn the inter-frame cross-attention features between the teacher network's input normal samples and inter-frame prototype features.

7. The image anomaly detection method based on knowledge transfer according to claim 1, characterized in that: The method of using the image anomaly detection model to obtain multi-scale feature differences of the image samples to be detected as image anomaly scores and detecting abnormal image samples includes: Obtain an image sample to be detected; Passing the image sample to be detected through the teacher network and the student network of the image anomaly detection model respectively to obtain multi-scale features; Calculating a difference map of the multi-scale features using a similarity metric; Performing interpolation fitting on the difference map of the multi-scale features, and adding the values ​​of corresponding positions of the difference map of the multi-scale features to obtain an image anomaly score map of the same scale as the image sample to be detected; If the image abnormality score map is less than the set threshold, it is determined to be a normal image; otherwise, it is determined to be an abnormal image.

8. An image anomaly detection system based on knowledge transfer, characterized in that: include: Anomaly detection model module: This module uses pre-trained teacher and student networks to build an initial anomaly detection model. It then uses the final anomaly detection model to obtain multi-scale feature differences of the image samples to be detected, which are used as image anomaly scores to detect abnormal image samples. A knowledge transfer module, which uses the acquired normal images as normal samples and transfers the normal sample knowledge in the pre-trained teacher network to the student network through intra-frame distillation; An inter-frame prototype feature library is used to store abnormal images with the largest representation difference between the pre-trained teacher network and the student network as near-abnormal samples to form an inter-frame prototype feature library; The student network training module uses the normal samples and the inter-frame prototype feature library to select intra-frame distillation and inter-frame distillation according to the probability strategy, and trains the normal sample representation ability and the near-abnormal sample representation ability of the student network respectively to obtain the final anomaly detection model.

9. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 7, or run the system according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 7, or to run the system according to claim 8.

Images