Spatial perception multi-view anomaly detection method based on meta-view representation

Abstract

The invention discloses a spatial perception multi-view anomaly detection method based on meta-view representation. The method comprises the following steps: obtaining multi-scale original features of a multi-view image of an input sample through a cascaded feature encoder; carrying out feature fusion on the multi-scale original features of the multi-view image, and obtaining meta-view representation through splicing, a full-connection network and a self-attention mechanism in sequence; performing a cross attention mechanism on the single-view image multi-scale features and the meta-view representation to obtain cross-view perceived single-view image multi-scale features, and splicing a plurality of cross-view perceived single-view image multi-scale features to obtain cross-view perceived multi-view image multi-scale features; according to the method, the cross-view perception multi-scale features of the multi-view image are compared with the original features of the multi-view image to perform anomaly detection, and the problem of view information loss of a traditional anomaly detection method is solved by extracting meta-view representation and performing cross-view perception, so that the accuracy of multi-view anomaly detection is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence anomaly detection, and particularly relates to a spatial perception multi-view anomaly detection method based on meta-view representation. BACKGROUND

[0002] Anomaly detection technology focuses on identifying sample anomalies to ensure quality and mitigate potential risks, and is gradually replacing traditional manual industrial product inspection and becoming the core means of modern factory quality control. In dynamic industrial scenarios, as the quality inspection requirements and acquisition equipment change, comprehensive detection of samples from multiple perspectives has become a new requirement. However, existing anomaly detection methods usually detect from a single perspective image, and lack the ability to effectively fuse the comprehensive information of multiple perspective images. This single perspective limits the global information capture of the anomaly detection method, resulting in information loss. In addition, when the sample has actual anomalies but the input single perspective image does not have anomalies, the sample-level prediction will be inaccurate. Based on the above problems, multi-view anomaly detection is proposed, which takes multiple perspective views of a single sample as input, and outputs whether each perspective image is abnormal and the abnormal area. Therefore, how to utilize other view information to assist single view image anomaly detection is the core of the multi-view anomaly detection task.

[0003] Current multi-view learning methods mainly fall into two categories. The first category is CNN-based fusion technology, which aggregates multi-view features through convolution. This category of methods tends to learn global consistent features, which can easily lead to smoothing and ignoring of abnormal signals. In addition, since convolution is performed on each view separately, the interaction between views is ignored. The second category is attention mechanism-based fusion methods. Attention mechanism-based fusion methods can capture inter-view dependencies, but existing methods cannot be directly applied to the multi-view anomaly detection task. For example, most methods are only suitable for fusing two views, so a multi-view fusion method needs to be designed for the multi-view anomaly detection task.

[0004] The core idea of existing multi-view anomaly detection schemes is to split images of different perspectives into image block features. By calculating the similarity between image block features under a single perspective and image block features of other perspectives, related features are found, and other view information is obtained through cross-attention mechanisms to assist single view anomaly detection. However, this method lacks modeling of global information, and as the number of associated features increases, the training time and resource requirements of the model will increase dramatically.

[0005] In summary, traditional multi-view fusion methods and single-view anomaly detection methods have the problem of being unable to effectively interact between views. Existing multi-view anomaly detection methods lack modeling of global information and have the problem of significantly increasing training time and resource requirements as the number of associated features increases. SUMMARY

[0006] Therefore, it is necessary to provide a spatial perception multi-view anomaly detection method based on meta-view representation in view of the above technical problems, which solves the problems that the traditional anomaly detection method cannot effectively interact between views and the training cost is high through the extraction of meta-view representation and the cross-view perception, and the method performs excellently in detecting any view anomaly, and improves the performance of multi-view anomaly detection and the accuracy of sample-level prediction.

[0007] The application discloses a spatial perception multi-view anomaly detection method based on meta-view representation, comprising the following steps: Step 1, obtaining multi-view image multi-scale original features of input samples through a cascaded feature encoder; Step 2, obtaining fused multi-view image multi-scale features by fusing the multi-view image multi-scale original features, and sequentially obtaining meta-view representation through splicing, a fully connected network and a self-attention mechanism; Step 3, sequentially performing flattening and dimension reshaping, adding position encoding, self-attention mechanism interaction, residual connection and layer normalization on single view image multi-scale features in the fused multi-view image multi-scale features, interacting with the meta-view representation through a cross-attention network, then performing residual connection and layer normalization, and sending into a feedforward neural network to obtain cross-view perception single view image multi-scale features, and splicing a plurality of cross-view perception single view image multi-scale features to obtain cross-view perception multi-view image multi-scale features; Step 4, comparing the cross-view perception multi-view image multi-scale features with the multi-view image original features to perform anomaly detection.

[0008] Further, regarding the step 2, the following steps are included: Step 201, fusing the multi-view image multi-scale original features to obtain fused multi-view image multi-scale features; Step 202, splicing the fused multi-view image multi-scale features under each view to obtain overall features of each sample at each scale; Step 203, matching the feature dimensions of the overall features of each sample at each scale with single view feature dimensions through a fully connected network; Step 204, obtaining meta-view representation through a self-attention mechanism for the matched overall features.

[0009] Further, regarding the step 201, the following steps are included: Step 2011: down-sampling the multi-view image multi-scale original features through a convolutional neural network to obtain fixed feature vector sizes; Step 2012: splicing the fixed feature vector sizes of the three scales, and sequentially up-sampling the fixed feature vector sizes of the three scales to obtain the fusion multi-view image multi-scale features of the three scales.

[0010] Further, regarding the step 204, the following steps are included: Step 2041: flattening the overall features after feature dimension matching into one-dimensional features; Step 2042: merging the spatial dimensions of the flattened overall features, and adding position encoding to the overall features after merging the spatial dimensions; Step 2043: obtaining global semantic representation through self-attention mechanism and residual connection to the overall features after adding position encoding, and then through layer normalization operation; Step 2044: obtaining meta-view representation after passing the global semantic representation through a feedforward neural network, and then through residual connection and layer normalization.

[0011] Further, regarding the step 3, the following steps are included: Step 301: flattening and dimension reshaping the single view image multi-scale features in the fusion multi-view image multi-scale features, adding position encoding after making the dimensions consistent with the meta-view representation; Step 302: pixel interaction through self-attention mechanism to the fusion single view image multi-scale features after adding position encoding, residual connection and layer normalization operation after the interaction; Step 303: interaction through cross-attention network between the single view image multi-scale features after layer normalization and the meta-view representation, then generating output representation integrating global information through residual connection and layer normalization; Step 304: sending the output representation integrating global information through layer normalization into a feedforward neural network to obtain cross-view perception single view image multi-scale features, and splicing multiple cross-view perception single view image multi-scale features to obtain cross-view perception multi-view image multi-scale features.

[0012] Further, regarding the step 4, the following steps are included: Step 401: reconstructing the cross-view perception multi-view image multi-scale features of each scale using a cascaded decoder; Step 402: in the network training stage, optimizing the entire network by minimizing the distance between the cross-view perception multi-view image multi-scale features of each scale and the multi-view image multi-scale original features through a loss function. Step 403: In the reasoning stage, the pixel-level anomaly scores of the encoder and the decoder at three scales are calculated by using the cosine similarity, and the maximum anomaly score in the same sample and different view images is taken as the sample-level anomaly score for anomaly detection.

[0013] Compared with the prior art, the present application has the beneficial effects that: By extracting the meta-view representation and performing cross-view perception, the problem that the traditional anomaly detection method cannot effectively perform inter-view interaction and the problem of high training cost are solved, and the method performs excellently in detecting anomalies of any view, thereby improving the performance of multi-view anomaly detection and the accuracy of sample-level prediction. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of the method of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0016] In order to better understand the technical content of the present embodiment, the terms involved in the present embodiment will be described first.

[0017] Anomaly detection: detecting whether there is an abnormal area in the actual sample.

[0018] Meta-view representation: a unified view representation that abstracts all view information.

[0019] Softmax: a normalized exponential function.

[0020] Position encoding: an operation of assigning position information to a group of vectors so that they can be distinguished from each other by position.

[0021] MLP: multi-layer perceptron, a kind of feedforward artificial neural network.

[0022] Concat: a vector splicing combination operation.

[0023] Figure 1 A flowchart of the embodiment of the present application is shown, and the present application discloses a spatial perception multi-view anomaly detection method based on meta-view representation, comprising the following steps: Step 1: Obtain multi-scale original features of the input sample multi-view image by passing it through a cascaded feature encoder. ; Step 2: Process the original features of the multi-view image at multiple scales. Multi-scale feature fusion is performed to obtain multi-scale features of the fused multi-view image, and meta-view representation is obtained through stitching, fully connected network and self-attention mechanism. ; Step 3: The single-view image multi-scale features in the fused multi-view image multi-scale features are sequentially flattened and dimensionally reshaped, positional encoding is added, self-attention mechanism interaction is performed, residual connection is performed, and layer normalization is performed. Then, they are interacted with the meta-view representation through a cross-attention network, followed by residual connection and layer normalization, and fed into a feedforward neural network to obtain cross-view perception single-view image multi-scale features. Multiple cross-view perception single-view image multi-scale features are concatenated to obtain cross-view perception multi-view image multi-scale features. Step 4: Combine the multi-scale features of the cross-view perception multi-view image with the original features of the multi-view image. Anomaly detection is performed by comparison.

[0024] The following describes the multi-view image anomaly detection process in detail with reference to specific embodiments.

[0025] Specifically, step 2 includes the following steps: Step 201: Extract multi-scale original features from multi-view images Multi-scale feature fusion is performed to obtain multi-scale features of the fused multi-view image; Step 202: Stitch together the multi-scale features of the fused multi-view image from each view to obtain the overall features of each sample at each scale. ; Step 203: Use a fully connected network to integrate the overall features at each scale. The feature dimensions are matched with the feature dimensions of a single view; Step 204: Analyze the overall features after matching. Obtaining metaview representation through self-attention mechanism .

[0026] Specifically, step 201 includes the following steps: Step 2011: Extract multi-scale original features from multi-view images A fixed feature vector size is obtained by downsampling using a convolutional neural network. ; The original multi-scale features of the multi-view image are:

[0027] wherein, represents the number of views, represents the height of the image, represents the width of the image, represents the feature dimension, represents different scales, is a real set; the multi-view image multi-scale original feature is down-sampled by a convolutional neural network to obtain a fixed feature vector size :

[0028] wherein, represents a convolution operation.

[0029] Step 2012: the fixed feature vector sizes of the three scales are spliced and are sequentially up-sampled to obtain three scales of fused multi-view image multi-scale features:

[0030] wherein, represents a splicing operation, respectively, the three scales of fused multi-view image multi-scale features, through multi-scale feature fusion, enhance the understanding of local details and global semantics.

[0031] Specifically, the step 204 includes the following steps: Step 2041: the overall feature after the feature dimension matching is flattened into a one-dimensional feature:

[0032] wherein, is a flattening operation, represents three different scales; Step 2042: the spatial dimension of the flattened overall feature is merged, and the overall feature after the merged spatial dimension is added with position encoding:

[0033] wherein, is a cosine position encoding, is a dimension reshaping operation; Step 2043: the overall feature after the position encoding is added is obtained through a self-attention mechanism and residual connection, and then through layer normalization operation to obtain global semantic representation​ :

[0034] wherein, denotes layer normalization, denotes self-attention mechanism, the self-attention mechanism specifically comprises the following steps: adding position encoding to the input overall feature performing linear projection to generate query (Query), key (Key) and value (Value) representations respectively through three learnable weight matrices, obtaining attention weights by dot product operation and scaling and softmax normalization between the query representation and the key representation, and performing weighted summation between the attention weights and the value representation to obtain an output result:

[0035] wherein, represents a learnable weight matrix, is a preset dimension of the query vector, and can be specifically represented as ; step 2044: obtaining the global semantic representation through a feedforward neural network, and then performing residual connection and layer normalization once to obtain a meta-view representation :

[0036] wherein, denotes a feedforward neural network, which is composed of two linear layers and one activation layer.

[0037] Specifically, the step 3 comprises the following steps: step 301: adding position encoding to the single-view image multi-scale feature in the fusion multi-view image multi-scale feature performing flattening and dimension reshaping operations, so that the feature dimension of the flattened and dimension reshaped single-view image multi-scale feature is consistent with the dimension of the meta-view representation :

[0038]

[0039] wherein, represents a view number; step 302: performing pixel interaction on the single-view image multi-scale feature added with position encoding through a self-attention mechanism, and then performing residual connection and layer normalization operations:

[0040] Step 303: Multi-scale features of a single view image after layer normalization and the metaview representation Interacting via a cross-attention network, and then generating an output representation that integrates global information through residual connections and layer normalization:

[0041] in, For a cross-attention network, the specific steps include: Query, key, and value representations are generated using three independent, learnable weight matrices. The query and key representations are multiplied by a dot product and then processed using a softmax function to obtain the attention weight distribution. Finally, the weights and value representations are weighted and fused to obtain the output.

[0042] in, Represents the learnable weight matrix. The dimension of the preset query vector can be specifically represented as: ; Step 304: The output representation integrating global information is normalized and fed into the feedforward neural network to obtain multi-scale features of a single-view image with cross-view perception. The multi-scale features of the multiple cross-view perception single-view images are stitched together to obtain the multi-scale features of the cross-view perception multi-view image.

[0043]

[0044] Further, step 4 involves comparing the multi-scale features of the cross-view perception multi-view image with the original features of the multi-view image for anomaly detection, including the following steps: Step 401: Reconstruct features at each scale using cascaded decoders:

[0045] in, To find the mean function, To find the variance function; Step 402: During the network training phase, the loss function is used... Minimize reconstruction features Multi-scale original features of multi-view images The distance is used to optimize the entire network, and the loss function is:

[0046] in, representing mean square error, representing different scales; Step 403: in the reasoning stage, the cosine similarity is used to calculate the pixel-level anomaly scores of the encoder and the decoder at three scales, and the maximum anomaly score in the same sample different view images is taken as the sample-level anomaly score for anomaly detection, and the greater the anomaly score, the greater the difference between the same sample different view images; the smaller the anomaly score, the smaller the difference between the same sample different view images.

[0047] In the present application, the data set used is a multi-view data set Real-IAD, Real-IAD contains 30 categories of samples such as bottle caps, button batteries, erasers, plastic nuts, magnetic tapes and zippers, each sample contains five view images. There are 99721 normal samples, of which 36465 are used for training, and the remaining 63256 are used for testing, and there are 51329 abnormal samples. The data set resolution is 2000-5000, and a total of 111 types of anomalies are included, with an average of 3.7 types of anomalies per category. Through the present application, multi-view information can be aggregated for anomaly detection, significantly improving the accuracy of sample-level anomaly detection.

[0048] The technical principle of the method of the present application is: the embodiment of the present application proposes a spatial perception multi-view anomaly detection method based on meta-view representation, which solves the problem that the traditional anomaly detection method cannot effectively interact between views and the problem of high training cost through the extraction of meta-view representation and cross-view perception. The method performs excellently when detecting any view anomaly, improves the performance of multi-view anomaly detection and the accuracy of sample-level prediction.

[0049] The technical features or steps of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features or steps in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0050] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

Claims

1. A spatial-aware multi-view anomaly detection method based on meta-view representation, characterized in that, The method comprises the following steps: Step 1: obtaining multi-view image multi-scale original features of input samples through a cascaded feature encoder; Step 2: performing multi-scale feature fusion on the multi-view image multi-scale original features to obtain fused multi-view image multi-scale features, and sequentially obtaining meta-view representations through splicing, a fully connected network and a self-attention mechanism; Step 3: sequentially performing flattening and dimension reshaping, adding position encoding, self-attention mechanism interaction, residual connection and layer normalization on single-view image multi-scale features in the fused multi-view image multi-scale features, interacting with the meta-view representations through a cross-attention network, then performing residual connection and layer normalization, and feeding into a feedforward neural network to obtain cross-view perception single-view image multi-scale features, and splicing a plurality of the cross-view perception single-view image multi-scale features to obtain cross-view perception multi-view image multi-scale features; Step 4: comparing the cross-view perception multi-view image multi-scale features with the multi-view image original features to perform anomaly detection.

2. The spatial-aware multi-view anomaly detection method based on meta-view representation according to claim 1, characterized in that, Regarding the step 2, the method comprises the following steps: Step 201: performing multi-scale feature fusion on the multi-view image multi-scale original features to obtain fused multi-view image multi-scale features; Step 202: splicing the fused multi-view image multi-scale features under each view to obtain overall features of each sample at each scale; Step 203: matching feature dimensions of the overall features of each sample at each scale with single-view feature dimensions through a fully connected network; Step 204: obtaining meta-view representations through a self-attention mechanism for the matched overall features.

3. The spatial-aware multi-view anomaly detection method based on meta-view representation according to claim 2, characterized in that, Regarding the step 201, the method comprises the following steps: Step 2011: down-sampling the multi-view image multi-scale original features through a convolutional neural network to obtain fixed feature vector sizes; Step 2012: splicing the fixed feature vector sizes of three scales, and sequentially up-sampling the fixed feature vector sizes of three scales to obtain the fused multi-view image multi-scale features of three scales.

4. The spatial-aware multi-view anomaly detection method based on meta-view representation according to claim 2, characterized in that, Regarding the step 204, the method comprises the following steps: Step 2041: flattening the overall features after matching feature dimensions into one-dimensional features; Step 2042: merging spatial dimensions of the flattened overall features, and adding position encoding to the overall features after merging spatial dimensions; Step 2043: obtaining global semantic representations through a self-attention mechanism and residual connection for the overall features after adding position encoding, and then performing layer normalization; Step 2044: obtaining meta-view representations through a feedforward neural network for the global semantic representations, and then performing residual connection and layer normalization once.

5. The spatially aware multi-view anomaly detection method based on meta-view representation of claim 4, wherein, Regarding the step 3, the method comprises the following steps: Step 301: flattening and dimension reshaping the single-view image multi-scale features in the fused multi-view image multi-scale features, so that the feature dimensions of the single-view image multi-scale features are consistent with the meta-view representation dimensions, and then adding position encoding; Step 302: Interacting the single-view image multi-scale features with pixel-to-pixel through self-attention mechanism, and then performing residual connection and layer normalization operation after the interaction; Step 303: Interacting the single-view image multi-scale features after layer normalization and the meta-view representation through cross-attention network, and then generating output representation integrating global information through residual connection and layer normalization; Step 304: Sending the output representation integrating global information to a feedforward neural network through layer normalization to obtain cross-view perception single-view image multi-scale features, and then splicing multiple cross-view perception single-view image multi-scale features to obtain cross-view perception multi-view image multi-scale features.

6. The spatial-aware multi-view anomaly detection method based on meta-view representation of claim 1, wherein, Regarding the step 4, the following steps are included: Step 401: Reconstructing the cross-view perception multi-view image multi-scale features of each scale by using a cascaded decoder; Step 402: In the network training stage, optimizing the entire network by minimizing the distance between the reconstructed cross-view perception multi-view image multi-scale features of each scale and the multi-view image multi-scale original features through a loss function; Step 403: In the inference stage, calculating the pixel-level anomaly scores of the encoder and decoder at three scales by using cosine similarity, and taking the maximum anomaly score in different view images of the same sample as the sample-level anomaly score for anomaly detection.

Images