Image anomaly detection method and system based on autoencoder feature difference

Through the autoencoder-based feature difference detection method, combining depth map and texture data, abnormal masks and sampling are generated, and feature reconstruction and segmentation network are trained, the problem of low abnormal area positioning and authenticity in the existing technology is solved, and the problem of low abnormal area positioning and authenticity is realized. More efficient detection of abnormal areas.

CN118799312BActive Publication Date: 2024-11-26QUANZHOU INST OF EQUIP MFG +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411267535.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-26
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

The existing image abnormality detection methods cannot effectively ensure the accurate positioning and reconstruction accuracy of abnormal areas in the semantic areas, and the synthesis of abnormal samples is relatively random and has low authenticity.

Method used

Using the feature difference detection method based on the autoencoder, the depth map is predicted through the Depth Anything model, the two-dimensional stacked fuzzy filter and Berlin noise generate anomaly mask, combined with texture data to sample abnormal pixels, train feature reconstruction networks and multi-scale segmentation networks , using the combination of cosine similarity loss and characteristic difference values ​​to improve the positioning and authenticity of the abnormal region.

Benefits of technology

It improves the positioning accuracy and synthesis authenticity of the abnormal area, alleviates the fine-grained problem in image reconstruction method, ensures the global consistency of the segmentation results, and intuitively represents the abnormal signal through the feature differences output by the encoder and decoder.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799312B_ABST
    Figure CN118799312B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of computer vision, and in particular to an image anomaly detection method and system based on autoencoder feature differences, S1: obtaining a training sample set #imgabs0# and a texture data set #imgabs1#; S2: inputting the training sample set #imgabs2# and the texture data set #imgabs3# into an anomaly generator to obtain an abnormal sample #imgabs4#; S3: inputting the training sample set #imgabs5# and the abnormal sample #imgabs6# into a feature reconstruction network for training to obtain normal features and reconstruction features; S4: respectively input the normal feature and the reconstructed feature into the multi-scale segmentation network to combine the feature differences and obtain the fused feature #imgabs7#; S5: input the fused feature #imgabs8# into the segmentation head network to determine the abnormal score #imgabs9#; S6: according to the preset rules, use the abnormal score #imgabs10# to obtain the image score, and output the abnormal score #imgabs11# and the image score; the present invention uses the depth information of the image to guide the abnormal synthesis area, improves the method of locating the abnormal area, and also improves the authenticity of the abnormal area synthesis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image anomaly detection method and system based on autoencoder feature differences. Background Art

[0002] Existing image anomaly detection methods often use unsupervised methods to train samples. Since the unsupervised method training process only includes normal samples, it is necessary to synthesize abnormal samples first, usually by randomly adding abnormal pixels at any position on the normal image. This method cannot guarantee that the abnormal area appears in the semantic area of ​​the image, and it is more random and less realistic. The autoencoder-based anomaly detection method mainly uses the autoencoder for image reconstruction, that is, it hopes that when an abnormal test image is input, the abnormal area in the output reconstructed image is reconstructed into normal pixels. The abnormal area can be located by comparing the input and output, but the existing method cannot guarantee the pixel accuracy after reconstruction. Summary of the invention

[0003] The object of the present invention is to provide an image anomaly detection method and system based on autoencoder feature differences, which can improve the interactivity between an abnormal synthesis area and a foreground semantic part in an image and improve reconstruction performance.

[0004] In order to achieve the above object, the present invention adopts such technical solution:

[0005] An image anomaly detection method based on autoencoder feature difference includes the following steps performed in sequence:

[0006] S1: Get training sample set and texture datasets ;

[0007] S2: The training sample set and the texture dataset Input into the anomaly generator to synthesize the anomaly in the foreground area of ​​the image and obtain the anomaly sample ;

[0008] The abnormal synthesis of the abnormal generator in step S2 includes the following steps:

[0009] S2-1: Use the Depth Anything model to predict the training sample set Depth map , depth map The color differences in the reflect the depth differences;

[0010] S2-2: Use a two-dimensional stacked blur filter to transform the depth map The boundary is expanded and the depth map is obtained by a preset threshold. Sample binary mask of , that is, the abnormal mask ;

[0011] S2-3: Using two-dimensional Perlin noise to obtain the initial anomaly mask , retain the initial anomaly mask With abnormal mask The mask area with intersection is used as the actual anomaly mask ;

[0012] S2-4: Use the following formula (1) to extract the texture data set Abnormal pixels Sampling of:

[0013] (1);

[0014] The abnormal pixels are converted into With training sample set Combine to get abnormal samples :

[0015] (2);

[0016] in, represents element-wise dot product, is the transparency of image fusion;

[0017] S3: The training sample set and the abnormal samples Input feature reconstruction network for training, the normal features extracted by the encoder of feature reconstruction network are , and , the reconstructed features output by the decoder of the feature reconstruction network are , and ;

[0018] S4: Input the normal features and the reconstructed features into the multi-scale segmentation network respectively to combine the feature differences to obtain the fusion features ;

[0019] S5: Fusion features Input anomaly score in the segmentation head network ;

[0020] S6: Using the abnormal score according to the preset rules Obtain image score and output anomaly score and image scores.

[0021] Preferably, in step S2-3, the initial abnormal mask With abnormal mask Perform an AND operation to obtain the mask image, and filter the mask image by the maximum connected domain to output the actual anomaly mask. .

[0022] Preferably, in step S3, a ResNet34 network pre-trained on the ImageNet dataset is used as the encoder of the feature reconstruction network with the last fully connected layer removed, and a reverse ResNet34 network with the same structure as the encoder is used as the decoder;

[0023] The training steps of the feature reconstruction network are as follows:

[0024] S3-1: The first, second and third layers of the encoder of the feature reconstruction network are respectively used for the training sample set Perform feature extraction and obtain the normal features as , and ,The normal features are stored in the encoder’s memory as the optimization target of the feature reconstruction network;

[0025] S3-2: The first, second, and third layers of the decoder of the feature reconstruction network respectively perform abnormal samples Perform feature reconstruction and output reconstructed features , and ;

[0026] The normal features and the reconstructed features are denoted as ;

[0027] S3-3: Cosine similarity is used to reduce the distance between the reconstructed features and the normal features, and the total reconstruction loss It is composed of three pairs of feature maps , , The cosine similarity loss of is calculated by summing up, and the specific calculation formula is shown in the following formula (3):

[0028] (3);

[0029] in, Represents the characteristics of a certain layer. , ;

[0030] For each The cosine similarity loss is given by and Characteristics in its channel dimension , The cosine loss is calculated by summing up. The following formula (4) is used for calculation:

[0031] (4);

[0032] in, Represents the characteristics of a channel. , C express The number of channels.

[0033] Preferably, the specific steps of combining feature differences in the multi-scale segmentation network in step S4 are as follows:

[0034] S4-1: Calculate the feature difference between the normal feature and the corresponding reconstructed feature using the following formula (5): (5);

[0035] in, ;

[0036] S4-2: Two neighborhood bilateral guided aggregation modules are used to combine multiple feature differences. and Through the first neighborhood bilateral guided aggregation module, the intermediate features are output ;

[0037] S4-3: and Send it to the second neighborhood bilateral guided aggregation module to obtain the final fusion feature .

[0038] Preferably, the segmentation head network uses anomaly score With real mask The mean absolute error and Sum to calculate the training loss of the feature segmentation network The specific process is as follows:

[0039] ;

[0040] ;

[0041] ;

[0042] in, The weight coefficient related to this category is set to 0.5. To adjust the parameters, set it to 4. Predict the correct probability for each pixel, It means that the horizontal axis is , the vertical axis is location, for exist The pixel anomaly score at this location, for exist The true mask for this position.

[0043] Preferably, the preset rule of step S6 is: Sort from largest to smallest and take the top 200 anomaly scores The average value of is taken as the image score.

[0044] An image anomaly detection system includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any of the above methods is implemented.

[0045] By adopting the above-mentioned design scheme, the beneficial effects of the present invention are as follows: the present invention uses the depth information of the image to guide the abnormal synthesis area, which improves the method of locating the abnormal area on the one hand, and improves the authenticity of the abnormal area synthesis on the other hand;

[0046] The present invention uses normal samples and abnormal samples to perform network training in a feature reconstruction network: fully utilizes the relationship between the internal features of the reconstruction network, and alleviates the fine-grained problem caused by image reconstruction in previous methods;

[0047] The present invention adopts multi-scale feature differences to perform abnormal area segmentation: multi-scale feature fusion ensures the global consistency of the segmentation results, and the feature differences obtained by subtracting the encoder and decoder outputs can intuitively represent abnormal signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the detection method of the present invention;

[0049] Figure 2 is a work flow chart of the exception generator of the present invention;

[0050] Figure 3 It is a structural schematic diagram of the feature reconstruction network of the present invention;

[0051] Figure 4 A training flow chart of a feature reconstruction network of the present invention;

[0052] Figure 5 It is a training flow chart of the multi-scale segmentation network of the present invention;

[0053] Figure 6 It is a structural schematic diagram of the neighborhood bilateral guidance aggregation module of the present invention;

[0054] Figure 7 It is a schematic diagram of the abnormal synthesis effect of the present invention;

[0055] Figure 8 It is a schematic diagram of the detection effect of the present invention;

[0056] Fig. 9 The training sample set of the present invention Depth map Schematic diagram of . DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] An image anomaly detection method based on autoencoder feature differences, such as Figure 1 As shown, the steps include the following steps performed in sequence:

[0059] S1: Get training sample set and texture datasets ;

[0060] In this embodiment, the training sample set The well-known dataset MVTec AD is used, which contains 3629 normal training images and 1725 test images, where the test images include various types of abnormal and normal samples. In this embodiment, the 3629 normal training images in the dataset MVTec AD are used as the training sample set. , 1725 test images in the dataset MVTec AD are used to test the image anomaly detection method to obtain the detection effect of the image anomaly detection method, such as Figure 8 shown.

[0061] The texture data set used in this embodiment It is a well-known data set DTD, or you can use your own texture map that is irrelevant to the task to build your own texture data set.

[0062] S2: training sample set and texture datasets Input to the exception generator, such as Figure 2 As shown, the abnormal synthesis of the image foreground area is performed to obtain abnormal samples ,like Figure 7 As shown, the anomaly generator includes a Depth Anything model, a two-dimensional stacked blur filter, and a two-dimensional Perlin noise;

[0063] In this embodiment, the exception synthesis of the exception generator includes the following steps:

[0064] S2-1: Use the Depth Anything model to predict the training sample set Depth map ,like Fig. 9 As shown, the depth map The grayscale value difference in reflects the depth difference. The smaller the grayscale value, the greater the depth. It can be seen that the grayscale value of the foreground area is significantly smaller than that of the background area, which also verifies the feasibility of using depth information to separate the background.

[0065] S2-2: In order to adapt to potential anomalies at the boundaries, a two-dimensional stack blur (StackBlur) filter is used to transform the depth map In this embodiment, the cv2.stackBlur function of the opencv library is used to implement the depth map The boundary is expanded and the depth map is obtained by a preset threshold. Sample binary mask of , that is, the abnormal mask In this embodiment, the method of presetting the threshold in step S2-2 is: The pixel range from 0 to 255 is normalized to 0 to 1, and the part greater than 0.3 is used as an abnormal mask. , that is, the threshold is 0.3. The threshold can be set according to actual needs.

[0066] S2-3: Use conventional two-dimensional Perlin noise to obtain the initial anomaly mask In this embodiment, according to the training sample set The height and width of the initial anomaly mask are set The size of the image is set, the noise ratio of the Perlin noise is preset, and a two-dimensional Perlin noise with a value between 0 and 1 is generated according to the calculation method of the Perlin noise. The two-dimensional Perlin noise is rotated to increase randomness, and a threshold is applied to the rotated two-dimensional Perlin noise to generate a binary image. If the noise value is greater than the threshold value 0.5, it is set to 1, otherwise it is set to 0. Finally, a shape and a training sample set are obtained. Consistent binary image, i.e. initial anomaly mask .

[0067] Keep the initial anomaly mask With abnormal mask The mask area with intersection is used as the actual anomaly mask In this embodiment, the initial abnormal mask With abnormal mask Perform an AND operation to obtain the mask image, and filter the mask image by the maximum connected domain to output the actual anomaly mask. ;

[0068] The maximum connected area filtering steps in this embodiment are: obtaining all valid connected areas according to the outline; storing the area of ​​each valid connected area; storing the index of the maximum area connected area; setting the pixel values ​​of all connected areas except the maximum area connected area to 0.

[0069] S2-4: Use the following formula (1) to extract the texture data set Abnormal pixels Sampling of:

[0070] (1);

[0071] The abnormal pixels are converted into With training sample set Combine to get abnormal samples :

[0072] (2);

[0073] in, represents element-wise dot product, is the transparency of image fusion, which is set to 0.5 in this embodiment; this abnormal synthesis method effectively improves randomness and authenticity.

[0074] S3: training sample set and abnormal samples Input feature reconstruction network for training, such as Figure 3 and Figure 4 As shown, the normal features extracted by the encoder of the feature reconstruction network are , and , the reconstructed features output by the decoder of the feature reconstruction network are , and ;

[0075] In this embodiment, a ResNet34 network pre-trained on the ImageNet dataset is used as the encoder of the feature reconstruction network by removing the last fully connected layer, and a reverse ResNet34 network with the same structure as the encoder is used as the decoder; the feature reconstruction network used in the present invention must be an autoencoder with a "symmetrical structure", and the resnet series and its variant models can be used to construct the feature reconstruction network. The use of resnet34 in this embodiment is based on the trade-off between speed and accuracy, and other choices can be made according to needs.

[0076] The training steps of the feature reconstruction network are as follows:

[0077] S3-1: The first, second and third layers of the encoder of the feature reconstruction network are respectively used for the training sample set Perform feature extraction and obtain the normal features as , and ,The normal features are stored in the encoder’s memory as the optimization target of the feature reconstruction network;

[0078] In this implementation, the input training sample set The dimensions (C, H, W) are (3, 256, 256);

[0079] S3-2: The first, second, and third layers of the decoder of the feature reconstruction network respectively perform abnormal samples Perform feature reconstruction and output reconstructed features , and ;

[0080] In this embodiment, the output reconstruction feature , and The sizes of are (64, 64, 64), (128, 32, 32) and (256, 16, 16) respectively; the normal features and the reconstructed features are recorded as ;

[0081] S3-3: Cosine similarity is used to reduce the distance between the reconstructed features and the normal features, and the total reconstruction loss It is composed of three pairs of feature maps , , The cosine similarity loss of is calculated by summing up, and the specific calculation formula is shown in the following formula (3):

[0082] (3);

[0083] in, Represents the characteristics of a certain layer. , ;

[0084] For each The cosine similarity loss is given by and Characteristics in its channel dimension , The cosine loss is calculated by summing up. The following formula (4) is used for calculation:

[0085] (4);

[0086] in, Represents the characteristics of a channel. , C express The number of channels.

[0087] S4: Figure 5 and Figure 6 As shown in Figure 2, the normal features and the reconstructed features are respectively input into the multi-scale segmentation network to combine the feature differences and obtain the fusion features. ;

[0088] The specific steps of the multi-scale segmentation network for feature difference combination are as follows:

[0089] S4-1: Calculate the feature difference between the normal feature and the corresponding reconstructed feature using the following formula (5): (5);

[0090] in, ;

[0091] S4-2: Two Neighborhood Bilateral Guided Aggregation (NBGA) modules are used to combine multiple feature differences, such as Figure 5 As shown, and Through the first neighborhood bilateral guided aggregation module, the intermediate features are output ;

[0092] S4-3: and Send it to the second neighborhood bilateral guided aggregation module to obtain the final fusion feature .

[0093] By combining two neighborhood bilateral guided aggregation modules, the weights of deep semantic features dynamically adjust shallow detail features, effectively suppressing irrelevant or noise features and making the model more robust.

[0094] S5: Fusion features Input anomaly score in the segmentation head network ; In this embodiment, to ensure simplicity, the segmentation head network is only composed of simple 3x3 convolutions and 1x1 convolutions.

[0095] Fusion Features After a convolution layer with a convolution kernel size of 3x3, the number of output feature channels is consistent with the input; after being processed by a batch normalization layer and a ReLU activation function, and finally passing through a convolution layer with a convolution kernel of 1x1, the number of feature channels is reduced from 4C to 1.

[0096] The segmentation head network uses anomaly score With real mask The mean absolute error and Sum to calculate the training loss of the feature segmentation network The specific process is as follows:

[0097] ;

[0098] ;

[0099] ;

[0100] in, The weight coefficient related to this category is set to 0.5. To adjust the parameters, set it to 4. Predict the correct probability for each pixel, It means that the horizontal axis is , the vertical axis is location, for exist The pixel anomaly score at this location, for exist The true mask at this location;

[0101] In this example, the training loss is calculated It is used to optimize the parameters of the segmentation head network using the optimizer so that the anomaly score As close to the real mask as possible The optimizer uses stochastic gradient descent (SGD) with a momentum of 0.9, a weight decay of 1e-4, and the initial learning rates of the reconstruction network and the segmentation network are 0.4 and 0.01, respectively.

[0102] S6: Using the abnormal score according to the preset rules Obtain image score and output anomaly score and image scores; in this embodiment, the abnormality score Sort from largest to smallest and take the top 200 anomaly scores The average value of is taken as the image score.

[0103] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image anomaly detection method based on autoencoder feature difference, characterized by: The process includes the following steps: S1: Get training sample set and texture datasets , where the training sample set The dimensions are (C, H, W); S2: The training sample set and the texture dataset Input into the anomaly generator to synthesize the anomaly in the foreground area of ​​the image and obtain the anomaly sample ; The abnormal synthesis of the abnormal generator in step S2 includes the following steps: S2-1: Use the Depth Anything model to predict the training sample set The depth map of the image, where the color difference reflects the depth difference; S2-2: Use a two-dimensional stacked blur filter to expand the boundary of the depth map, and obtain the sample binary mask of the depth map through a preset threshold , that is, the abnormal mask ; S2-3: Using two-dimensional Perlin noise to obtain the initial anomaly mask , retain the initial anomaly mask With abnormal mask The mask area with intersection is used as the actual anomaly mask ; S2-4: Use the following formula (1) to extract the texture data set Abnormal pixels Sampling of: (1); The abnormal pixels are converted into With training sample set Combine to get abnormal samples : (2); in, represents element-wise dot product, is the transparency of image fusion; S3: The training sample set and the abnormal samples Input feature reconstruction network for training, the normal features extracted by the encoder of feature reconstruction network are , and , the reconstructed features output by the decoder of the feature reconstruction network are , and ; In step S3, a ResNet34 network pre-trained on the ImageNet dataset is used as the encoder of the feature reconstruction network with the last fully connected layer removed, and a reverse ResNet34 network with the same structure as the encoder is used as the decoder; The training steps of the feature reconstruction network are as follows: S3-1: The first, second and third layers of the encoder of the feature reconstruction network are respectively used for the training sample set Perform feature extraction and obtain the normal features as , and ,The normal features are stored in the encoder’s memory as the optimization target of the feature reconstruction network; S3-2: The first, second, and third layers of the decoder of the feature reconstruction network respectively perform abnormal samples Perform feature reconstruction and output reconstructed features , and ; The normal features and the reconstructed features are denoted as , ,in, Represents the characteristics of a certain layer. ; S3-3: Cosine similarity is used to reduce the distance between the reconstructed features and the normal features, and the total reconstruction loss It is composed of three pairs of feature maps , , The cosine similarity loss of is calculated by summing up, and the specific calculation formula is shown in the following formula (3): (3); in, ; For each The cosine similarity loss is given by and Characteristics in its channel dimension , The cosine similarity loss is calculated by summing up. The following formula (4) is used for calculation: (4); in, Represents the characteristics of a channel. , C express The number of channels; S4: Input the normal features and the reconstructed features into a multi-scale feature segmentation network to combine feature differences and obtain fusion features ; The specific steps of the multi-scale feature segmentation network in step S4 for combining feature differences are as follows: S4-1: Calculate the feature difference between the normal feature and the corresponding reconstructed feature using the following formula (5): (5); in, ; S4-2: Two neighborhood bilateral guided aggregation modules are used to combine multiple feature differences. and Through the first neighborhood bilateral guided aggregation module, the intermediate features are output ; S4-3: and Send it to the second neighborhood bilateral guided aggregation module to obtain the final fusion feature ; S5: Fusion features Input the abnormality score in the segmentation head network of the multi-scale feature segmentation network ; S6: Using the abnormal score according to the preset rules Obtain image score and output anomaly score and image scores.

2. The image anomaly detection method based on autoencoder feature difference according to claim 1, characterized in that: In step S2-3, the initial abnormal mask With abnormal mask Perform an AND operation to obtain the mask image, and filter the mask image by the maximum connected domain to output the actual anomaly mask. .

3. The image anomaly detection method based on autoencoder feature difference according to claim 1, characterized in that: The preset rule of step S6 is: Sort from largest to smallest and take the top 200 anomaly scores The average value of is taken as the image score.

4. An image anomaly detection system, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the computer program, the image anomaly detection method based on autoencoder feature differences described in any one of claims 1 to 3 is implemented.