A Spatial Awareness Multi-View Anomaly Detection Method Based on Meta-View Representation

Abstract

This invention discloses a spatially aware multi-view anomaly detection method based on meta-view representation. The method includes: obtaining multi-scale original features of the multi-view images of the input samples through a cascaded feature encoder; fusing the original multi-scale features of the multi-view images and sequentially passing them through concatenation, a fully connected network, and a self-attention mechanism to obtain meta-view representations; passing the multi-scale features of a single view image and the meta-view representations through a cross-attention mechanism to obtain cross-view aware single-view image multi-scale features; concatenating multiple cross-view aware single-view image multi-scale features to obtain cross-view aware multi-view image multi-scale features; and comparing the cross-view aware multi-view image multi-scale features with the original features of the multi-view images for anomaly detection. By extracting meta-view representations and performing cross-view awareness, this method solves the problem of missing view information in traditional anomaly detection methods, thereby improving the accuracy of multi-view anomaly detection.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence anomaly detection, and in particular relates to a spatially perceptive multi-view anomaly detection method based on meta-view representation. Background Technology

[0002] Anomaly detection technology focuses on identifying sample anomalies to ensure quality and mitigate potential risks, and is gradually becoming...

[0003] Replacing traditional manual inspection of industrial products, multi-view anomaly detection has become a core means of quality control in modern factories. In dynamically changing industrial scenarios, with changes in quality inspection requirements and acquisition equipment, comprehensive inspection of samples from multiple perspectives has become a new demand. However, existing anomaly detection methods typically detect anomalies from a single-view image, lacking the ability to effectively fuse comprehensive information from multiple view images. This single perspective limits the capture of global information by anomaly detection methods, leading to information gaps. Furthermore, when a sample has actual anomalies but the input single-view image does not, it can lead to inaccurate sample-level predictions. Based on these problems, multi-view anomaly detection is proposed. It uses multiple views of a single sample as input, and the output is still whether each view image is abnormal and the abnormal region. Therefore, how to utilize information from other views to assist in anomaly detection of single-view images is the core of the multi-view anomaly detection task.

[0004] Current multi-view learning methods mainly fall into two categories. The first category is CNN-based fusion techniques, which aggregate multi-view features through convolution. These methods tend to learn globally consistent features, which can easily lead to the smoothing and ignoring of anomalous signals. Furthermore, because convolution is performed on each view individually, the interactions between views are ignored. The second category is attention-based fusion methods. Attention-based fusion methods can capture dependencies between views, but existing methods cannot be directly applied to multi-view anomaly detection tasks. For example, most methods are only suitable for fusing two views. Therefore, it is necessary to design multi-view fusion methods specifically for multi-view anomaly detection tasks.

[0005] The core idea of ​​existing multi-view anomaly detection schemes is to segment images from different perspectives into image patch features. By calculating the similarity between image patch features from a single perspective and image patch features from other perspectives, related features are found. Information from other views is obtained through a cross-attention mechanism to assist in anomaly detection in a single view. However, this method lacks modeling of global information, and the training time and resource requirements of the model increase dramatically as the number of associated features increases.

[0006] In summary, traditional multi-view fusion methods and single-view anomaly detection methods cannot effectively facilitate interaction between views. Existing multi-view anomaly detection methods lack modeling of global information, and training time and resource requirements increase significantly with the number of associated features. Summary of the Invention

[0007] In view of this, it is necessary to provide a spatially perceptive multi-view anomaly detection method based on meta-view representation to address the above-mentioned technical problems. This method solves the problems of traditional anomaly detection methods being unable to effectively perform inter-view interaction and having high training costs by extracting meta-view representation and performing cross-view perception. This method performs well in detecting anomalies in any view, improving the performance of multi-view anomaly detection and the accuracy of sample-level prediction.

[0008] This invention discloses a spatially aware multi-view anomaly detection method based on meta-view representation, comprising the following steps:

[0009] Step 1: Obtain multi-scale original features of the multi-view image by passing the multi-view image of the input sample through a cascaded feature encoder;

[0010] Step 2: Multi-scale feature fusion is performed on the original multi-scale features of the multi-view image to obtain fused multi-view image multi-scale features. The fused multi-view image multi-scale features are then sequentially processed through stitching, a fully connected network, and a self-attention mechanism to obtain meta-view representations.

[0011] Step 3: The single-view image multi-scale features in the fused multi-view image multi-scale features are sequentially flattened and dimensionally reshaped, positional encoding is added, self-attention mechanism interaction is performed, residual connection is performed, and layer normalization is performed. Then, they are interacted with the meta-view representation through a cross-attention network, followed by residual connection and layer normalization, and fed into a feedforward neural network to obtain cross-view perception single-view image multi-scale features. Multiple cross-view perception single-view image multi-scale features are concatenated to obtain cross-view perception multi-view image multi-scale features.

[0012] Step 4: Compare the multi-scale features of the cross-view perception multi-view image with the original features of the multi-view image to perform anomaly detection.

[0013] Furthermore, regarding step 2, it includes the following steps:

[0014] Step 201: Perform multi-scale feature fusion on the original multi-scale features of the multi-view image to obtain fused multi-scale features of the multi-view image;

[0015] Step 202: The multi-scale features of the fused multi-view image under each view are stitched together to obtain the overall features of each sample at each scale;

[0016] Step 203: Match the feature dimensions of the overall features of each sample at each scale with the feature dimensions of the single view using a fully connected network;

[0017] Step 204: Obtain the meta-view representation of the matched overall features through a self-attention mechanism.

[0018] Furthermore, regarding step 201, it includes the following steps:

[0019] Step 2011: The original multi-scale features of the multi-view image are downsampled using a convolutional neural network to obtain a fixed feature vector size;

[0020] Step 2012: Concatenate the fixed feature vector sizes of the three scales, and then upsample the fixed feature vector sizes of the three scales in sequence to obtain the multi-scale features of the fused multi-view image at the three scales.

[0021] Furthermore, step 204 includes the following steps:

[0022] Step 2041: Flatten the overall features after feature dimension matching into one-dimensional features;

[0023] Step 2042: Merge the spatial dimensions of the flattened overall features and add positional codes to the overall features after merging spatial dimensions;

[0024] Step 2043: The overall features after adding position encoding are processed through a self-attention mechanism and residual connections, and then the global semantic representation is obtained through layer normalization.

[0025] Step 2044: After passing the global semantic representation through a feedforward neural network, it is then subjected to a residual connection and layer normalization to obtain the meta-view representation.

[0026] Furthermore, regarding step 3, it includes the following steps:

[0027] Step 301: Flatten and reshape the multi-scale features of the individual view images in the fused multi-view image multi-scale features to make them consistent with the dimensions of the meta-view representation, and then add position encoding;

[0028] Step 302: The multi-scale features of the fused single view image after adding position encoding are interacted between pixels through a self-attention mechanism, and residual connection and layer normalization operations are performed after the interaction.

[0029] Step 303: The multi-scale features of the single view image after layer normalization and the meta-view representation are interacted through a cross-attention network, and then an output representation integrating global information is generated through residual connections and layer normalization;

[0030] Step 304: The output representation of the integrated global information is normalized by layers and fed into a feedforward neural network to obtain multi-scale features of a single-view image with cross-view perception. Multiple multi-scale features of a single-view image with cross-view perception are concatenated to obtain multi-scale features of a multi-view image with cross-view perception.

[0031] Furthermore, regarding step 4, it includes the following steps:

[0032] Step 401: Reconstruct the multi-scale features of the cross-view perception multi-view image at each scale using a cascaded decoder.

[0033] Step 402: During the network training phase, the entire network is optimized by minimizing the distance between the reconstructed multi-scale features of the cross-view perception multi-view image at each scale and the original multi-scale features of the multi-view image by using a loss function.

[0034] Step 403: During the inference phase, the encoder and decoder are calculated using cosine similarity at three scales to obtain pixel-level anomaly scores. The maximum anomaly score in the same sample from different viewpoints is used as the sample-level anomaly score for anomaly detection.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] By extracting meta-view representations and performing cross-view perception, this method solves the problems of traditional anomaly detection methods being unable to effectively perform inter-view interactions and having high training costs. This method performs well in detecting anomalies in any view, improving the performance of multi-view anomaly detection and the accuracy of sample-level prediction. Attached Figure Description

[0037] Figure 1 A flowchart illustrating the implementation method of the present invention is shown. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0039] To better understand the technical content of this embodiment, the terms involved in this embodiment will be explained first.

[0040] Anomaly detection: Detects whether there are abnormal regions in the actual sample.

[0041] Metaview Representation: A unified view representation that extracts all view information.

[0042] Softmax: Normalized exponential function.

[0043] Position encoding: An operation that assigns positional information to a set of vectors so that they can be distinguished from each other by their position.

[0044] MLP: Multilayer Perceptron, a type of feedforward artificial neural network.

[0045] Concat: A vector concatenation and combination operation.

[0046] Figure 1 The flowchart of an embodiment of the present invention is shown. The present invention discloses a spatially aware multi-view anomaly detection method based on meta-view representation, comprising the following steps:

[0047] Step 1: Obtain multi-scale original features of the input sample multi-view image by passing it through a cascaded feature encoder. ;

[0048] Step 2: Process the original features of the multi-view image at multiple scales. Multi-scale feature fusion is performed to obtain multi-scale features of the fused multi-view image, and meta-view representation is obtained through stitching, fully connected network and self-attention mechanism. ;

[0049] Step 3: The single-view image multi-scale features in the fused multi-view image multi-scale features are sequentially flattened and dimensionally reshaped, positional encoding is added, self-attention mechanism interaction is performed, residual connection is performed, and layer normalization is performed. Then, they are interacted with the meta-view representation through a cross-attention network, followed by residual connection and layer normalization, and fed into a feedforward neural network to obtain cross-view perception single-view image multi-scale features. Multiple cross-view perception single-view image multi-scale features are concatenated to obtain cross-view perception multi-view image multi-scale features.

[0050] Step 4: Combine the multi-scale features of the cross-view perception multi-view image with the original features of the multi-view image. Anomaly detection is performed by comparison.

[0051] The following describes the multi-view image anomaly detection process in detail with reference to specific embodiments.

[0052] Specifically, step 2 includes the following steps:

[0053] Step 201: Extract multi-scale original features from multi-view images Multi-scale feature fusion is performed to obtain multi-scale features of the fused multi-view image;

[0054] Step 202: Stitch together the multi-scale features of the fused multi-view image from each view to obtain the overall features of each sample at each scale. ;

[0055] Step 203: Use a fully connected network to integrate the overall features at each scale. The feature dimensions are matched with the feature dimensions of a single view;

[0056] Step 204: Analyze the overall features after matching. Obtaining metaview representation through self-attention mechanism .

[0057] Specifically, step 201 includes the following steps:

[0058] Step 2011: Extract multi-scale original features from multi-view images A fixed feature vector size is obtained by downsampling using a convolutional neural network. ;

[0059] The original multi-scale features of the multi-view image are:

[0060]

[0061] in, Represents the number of views. Represents the height of the image. Represents the width of the image. Represents feature dimension, Representing different scales. It is the set of real numbers;

[0062] Multi-scale original features of the multi-view image A fixed feature vector size is obtained by downsampling using a convolutional neural network. :

[0063]

[0064] in, This indicates a convolution operation.

[0065] Step 2012: Fix the feature vector size at three scales. The data is concatenated, and then sequentially processed using fixed feature vector sizes at three different scales. Upsampling was performed to obtain multi-scale features of the fused multi-view image at three different scales:

[0066]

[0067] in, This indicates a splicing operation. The multi-scale features of the multi-view image are fused at three scales, and the understanding of local details and global semantics is enhanced by multi-scale feature fusion.

[0068] Specifically, step 204 includes the following steps:

[0069] Step 2041: Match the overall features after feature dimensions Features of flattening into one dimension:

[0070]

[0071] in, For flattening operation, Representing three different scales;

[0072] Step 2042: Flatten the overall features The spatial dimensions are merged, and the overall characteristics after merging the spatial dimensions are analyzed. Add location encoding:

[0073]

[0074] in, Cosine position encoding, For dimensional reshaping operations;

[0075] Step 2043: Analyze the overall features after adding location encoding. Global semantic representation is obtained by using a self-attention mechanism and residual connections, followed by layer normalization. :

[0076]

[0077] in, Representation layer normalization, This indicates the self-attention mechanism.

[0078] The self-attention mechanism specifically includes the following steps:

[0079] Overall features after adding position encoding to the input Linear projection is performed, and query, key, and value representations are generated using three learnable weight matrices. The query and key representations are multiplied by a dot product and then scaled and normalized using softmax to obtain attention weights. The attention weights are then weighted and summed with the value representations to obtain the output.

[0080]

[0081] in, Represents the learnable weight matrix. The dimension of the preset query vector can be specifically represented as: ;

[0082] Step 2044: The global semantic representation After passing through a feedforward neural network, a residual connection and layer normalization are then performed to obtain the meta-view representation. :

[0083]

[0084] in, This represents a feedforward neural network, consisting of two linear layers and one activation layer.

[0085] Specifically, step 3 includes the following steps:

[0086] Step 301: Combine the multi-scale features of individual view images from the fused multi-view image multi-scale features. Perform flattening and dimensionality reshaping operations to enable multi-scale features in a single view image after flattening and dimensionality reshaping. Feature dimensions and the metaview representation After ensuring the dimensions are consistent, add positional encoding:

[0087]

[0088]

[0089] in, Represents the view number;

[0090] Step 302: Multi-scale features of a single view image with added positional encoding are processed using a self-attention mechanism. Perform inter-pixel interactions, followed by residual connections and layer normalization:

[0091]

[0092] Step 303: Multi-scale features of a single view image after layer normalization and the metaview representation Interacting via a cross-attention network, and then generating an output representation that integrates global information through residual connections and layer normalization:

[0093]

[0094] in, For a cross-attention network, the specific steps include:

[0095] Query, key, and value representations are generated using three independent, learnable weight matrices. The query and key representations are multiplied by a dot product and then processed using a softmax function to obtain the attention weight distribution. Finally, the weights and value representations are weighted and fused to obtain the output.

[0096]

[0097] in, Represents the learnable weight matrix. The dimension of the preset query vector can be specifically represented as: ;

[0098] Step 304: The output representation integrating global information is normalized and fed into the feedforward neural network to obtain multi-scale features of a single-view image with cross-view perception. The multi-scale features of the multiple cross-view perception single-view images are stitched together to obtain the multi-scale features of the cross-view perception multi-view image.

[0099]

[0100] Further, step 4 involves comparing the multi-scale features of the cross-view perception multi-view image with the original features of the multi-view image for anomaly detection, including the following steps:

[0101] Step 401: Reconstruct features at each scale using cascaded decoders:

[0102]

[0103] in, To find the mean function, To find the variance function;

[0104] Step 402: During the network training phase, the loss function is used... Minimize reconstruction features Multi-scale original features of multi-view images The distance is used to optimize the entire network, and the loss function is:

[0105]

[0106] in, Represents the mean square error. Representing different scales;

[0107] Step 403: During the inference phase, the encoder and decoder are calculated at the pixel level anomaly score on three scales using cosine similarity. The largest anomaly score among the images of the same sample from different viewpoints is used as the sample-level anomaly score for anomaly detection. The larger the anomaly score, the greater the difference between the images of the same sample from different viewpoints; the smaller the anomaly score, the smaller the difference between the images of the same sample from different viewpoints.

[0108] In this invention, the dataset used is the Real-IAD multi-view dataset. Real-IAD contains samples in 30 categories, such as bottle caps, button batteries, erasers, plastic nuts, magnetic tapes, and zippers. Each sample contains images from five views. There are 99,721 normal samples, of which 36,465 are used for training and the remaining 63,256 are used for testing. There are 51,329 anomalous samples. The dataset resolution is 2000-5000, containing 111 anomaly types, with an average of 3.7 anomaly types per category. Through this invention, multi-view information can be aggregated for anomaly detection, significantly improving the accuracy of sample-level anomaly detection.

[0109] The technical principle of the method of the present invention is as follows: The present invention proposes a spatially perceptive multi-view anomaly detection method based on meta-view representation. By extracting meta-view representation and performing cross-view perception, it solves the problems of traditional anomaly detection methods being unable to effectively perform inter-view interaction and high training costs. This method performs well in detecting anomalies in any view, improving the performance of multi-view anomaly detection and the accuracy of sample-level prediction.

[0110] The technical features or steps of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features or steps in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A spatially perceptive multi-view anomaly detection method based on meta-view representation, characterized in that, Includes the following steps: Step 1: Obtain multi-scale original features of the multi-view image by passing the multi-view image of the input sample through a cascaded feature encoder; Step 2: Perform multi-scale feature fusion on the original multi-scale features of the multi-view image to obtain fused multi-view image multi-scale features. The fused multi-view image multi-scale features are then sequentially processed through stitching, a fully connected network, and a self-attention mechanism to obtain meta-view representations. Step 3: The single-view image multi-scale features in the fused multi-view image multi-scale features are sequentially flattened and dimensionally reshaped, positional encoding is added, self-attention mechanism interaction is performed, residual connection is performed, and layer normalization is performed. Then, they are interacted with the meta-view representation through a cross-attention network, followed by residual connection and layer normalization, and fed into a feedforward neural network to obtain cross-view perception single-view image multi-scale features. Multiple cross-view perception single-view image multi-scale features are concatenated to obtain cross-view perception multi-view image multi-scale features. Step 4: Compare the multi-scale features of the cross-view perception multi-view image with the original multi-scale features of the multi-view image to perform anomaly detection.

2. The spatially aware multi-view anomaly detection method based on meta-view representation according to claim 1, characterized in that, Regarding step 2, it includes the following steps: Step 201: Perform multi-scale feature fusion on the original multi-scale features of the multi-view image to obtain fused multi-scale features of the multi-view image; Step 202: The multi-scale features of the fused multi-view image under each view are stitched together to obtain the overall features of each sample at each scale; Step 203: Match the feature dimensions of the overall features of each sample at each scale with the feature dimensions of the single view using a fully connected network; Step 204: Obtain the meta-view representation of the matched overall features through a self-attention mechanism.

3. The spatially perceptive multi-view anomaly detection method based on meta-view representation according to claim 2, characterized in that, Regarding step 201, it includes the following steps: Step 2011: The original multi-scale features of the multi-view image are downsampled using a convolutional neural network to obtain a fixed feature vector size; Step 2012: Concatenate the fixed feature vector sizes of the three scales, and then upsample the fixed feature vector sizes of the three scales in sequence to obtain the multi-scale features of the fused multi-view image at the three scales.

4. The spatially perceptive multi-view anomaly detection method based on meta-view representation according to claim 2, characterized in that, Regarding step 204, it includes the following steps: Step 2041: Flatten the overall features after feature dimension matching into one-dimensional features; Step 2042: Merge the spatial dimensions of the flattened overall features and add positional codes to the overall features after merging spatial dimensions; Step 2043: The overall features after adding position encoding are processed through a self-attention mechanism and residual connections, and then the global semantic representation is obtained through layer normalization. Step 2044: After passing the global semantic representation through a feedforward neural network, it is then subjected to a residual connection and layer normalization to obtain the meta-view representation.

5. The spatially aware multi-view anomaly detection method based on meta-view representation according to claim 4, characterized in that, Regarding step 3, it includes the following steps: Step 301: Flatten and reshape the multi-scale features of a single view image in the fused multi-view image multi-scale features, so that the feature dimension of the single view image multi-scale features is consistent with the meta-view representation dimension, and then add position encoding. Step 302: The multi-scale features of the single view image after adding position encoding are interacted between pixels through a self-attention mechanism, and residual connection and layer normalization operations are performed after the interaction. Step 303: The multi-scale features of the single view image after layer normalization and the meta-view representation are interacted through a cross-attention network, and then an output representation integrating global information is generated through residual connections and layer normalization; Step 304: The output representation of the integrated global information is normalized by layers and fed into a feedforward neural network to obtain multi-scale features of a single-view image with cross-view perception. Multiple multi-scale features of a single-view image with cross-view perception are concatenated to obtain multi-scale features of a multi-view image with cross-view perception.

6. The spatially aware multi-view anomaly detection method based on meta-view representation according to claim 1, characterized in that, Regarding step 4, it includes the following steps: Step 401: Reconstruct the multi-scale features of the cross-view perception multi-view image at each scale using a cascaded decoder. Step 402: During the network training phase, the entire network is optimized by minimizing the distance between the reconstructed multi-scale features of the cross-view perception multi-view image at each scale and the original multi-scale features of the multi-view image by using a loss function. Step 403: During the inference phase, the encoder and decoder are calculated using cosine similarity at three scales to obtain pixel-level anomaly scores. The maximum anomaly score in the same sample from different viewpoints is used as the sample-level anomaly score for anomaly detection.

Images