A Dual-Branch Image Anomaly Detection Method, Device, Equipment and Medium

Abstract

The present invention discloses a dual-branch image anomaly detection method, apparatus, device, and medium. The method includes: obtaining an image to be detected and performing preprocessing on the image to be detected; inputting the preprocessed image to be detected into a trained anomaly detection model to obtain an anomaly score; determining whether the image to be detected is an abnormal image or a normal image according to the anomaly score; wherein the training of the anomaly detection model includes: obtaining a preset number of sample images, performing rotation expansion and preprocessing on the sample images; constructing an anomaly detection model, which includes a visible anomaly branch and an invisible anomaly branch; training the anomaly detection model with the sample images after rotation expansion and preprocessing. The present invention can have good adaptability to different types of industrial images, can efficiently complete the anomaly detection task, and has a low computational complexity at the same time.

Description

Technical Field

[0001] The present invention relates to a dual-branch image anomaly detection method, device, equipment and medium, belonging to the technical field of image detection. Background Art

[0002] Image anomaly detection is a popular research topic in the field of computer vision, with high research significance and application value, and can be used in fields such as industrial appearance defect detection, medical image analysis, and hyperspectral image processing. With the development of neural network technology, more and more algorithm models are applied to image anomaly detection tasks, including industrial image anomaly detection.

[0003] In industrial production activities, a part of anomaly samples can be obtained. However, due to the uncertainty and randomness of anomalies, these existing anomaly samples usually only account for a small part of all anomalies. Existing anomaly detection algorithms are divided into two categories: supervised learning and unsupervised learning. Supervised learning uses a part of the existing anomaly samples, but rarely classifies these samples themselves, while unsupervised learning does not use the existing anomaly samples. By observing the anomaly samples, the anomaly samples can be roughly divided into two categories: samples with visible anomalies and samples with invisible anomalies. Among them, visible anomalies include anomaly images with obvious defects or damages, and invisible anomalies are images without obvious defects but with large differences from normal samples. Most algorithms do not consider classifying anomaly images into the above two categories, so the effect is often better only for a certain type of anomaly. In summary, there are some defects in the current anomaly detection methods: (1) Low detection accuracy due to low adaptability, and easy to misdetect; (2) Large amount of calculation. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art, and provide a dual-branch image anomaly detection method, device, equipment and medium to solve the technical problems of low accuracy and large amount of calculation of the current industrial image anomaly detection model.

[0005] To achieve the above purpose, the present invention is implemented by the following technical solutions:

[0006] In the first aspect, the present invention provides a dual-branch image anomaly detection method, including:

[0007] Obtain the image to be detected, and preprocess the image to be detected;

[0008] Input the preprocessed image to be detected into the trained anomaly detection model to obtain an anomaly score;

[0009] Judge whether the image to be detected is an anomaly image or a normal image according to the anomaly score;

[0010] Among them, the training of the anomaly detection model includes:

[0011] Obtain a preset number of sample images, and perform rotation expansion and preprocessing on the sample images;

[0012] Construct an anomaly detection model, where the anomaly detection model includes a visible anomaly branch and an invisible anomaly branch;

[0013] Train the anomaly detection model with the sample images after rotation expansion and preprocessing.

[0014] Optionally, the preprocessing includes resizing the image to 448×448.

[0015] Optionally, the judging whether the image to be detected is an abnormal image or a normal image according to the anomaly score includes:

[0016] If the anomaly score of the image to be detected is greater than or equal to 0.5 and less than or equal to 1, then the image to be detected is an abnormal image;

[0017] If the anomaly score of the image to be detected is greater than or equal to 0 and less than 0.5, then the image to be detected is a normal image.

[0018] Optionally, the rotation expansion of the sample images includes:

[0019] Rotate the sample images by 90°, 180°, and 270°, and save each rotated sample image.

[0020] Optionally, the training of the anomaly detection model with the sample images after rotation expansion and preprocessing includes:

[0021] Feed the sample images after rotation expansion and preprocessing into the visible anomaly branch to obtain an anomaly score F1;

[0022] Feed the sample images after rotation expansion and preprocessing into the invisible anomaly branch to obtain an anomaly score F2;

[0023] Perform global average pooling on the anomaly scores F1 and F2 respectively, and add the results of global average pooling to obtain z 1 ;

[0024] Reduce the dimension of z 1 to 1 through a fully connected layer to obtain z 2 , and calculate the attention of the visible anomaly branch and the invisible anomaly branch based on z 2 :

[0025]

[0026]

[0027] where \(l_1\) and \(l_2\) are the attentions of the visible anomaly branch and the invisible anomaly branch respectively, \(L_1\), \(H_1\) are the weight parameters of the visible anomaly branch, and \(L_2\), \(H_2\) are the weight parameters of the invisible anomaly branch;

[0028] Calculate the final anomaly score \(S\) based on the anomaly scores \(F_1\), \(F_2\) and the attentions \(l_1\), \(l_2\):

[0029] \(S = norm(F_1\times l_1 + F_2\times l_2)\)

[0030] where \(norm\) is the normalization function in \([0, 1]\);

[0031] Calculate the model loss \(Loss\) based on the anomaly score \(S\):

[0032] \(Loss=(1 - y)|D(S)|+ymax(0, a - D(S))\)

[0033]

[0034] where \(\mu_{\gamma}\), \(\delta_{\gamma}\) are the mean and standard deviation of the obtained anomaly scores; when \(D(S)\in[0.5, 1]\), \(y = 1\), when \(D(S)\in[0, 0.5)\), \(y = 0\); \(a\) is the preset confidence interval parameter;

[0035] Update the weight parameters \(L_1\), \(H_1\), \(L_2\), \(H_2\) according to the model loss \(Loss\) to obtain the updated anomaly detection model;

[0036] Repeat the above steps until the model loss \(Loss\) converges or reaches the preset maximum number of iterations to obtain the trained anomaly detection model.

[0037] Optionally, the step of sending the rotation-expanded and preprocessed sample image into the visible anomaly branch to obtain the anomaly score \(F_1\) includes:

[0038] Extract features from the rotation-expanded and preprocessed sample image through a convolutional layer to generate a first feature map;

[0039] Divide the first feature map into multiple first feature blocks according to a preset size, and obtain the pixel values of each first feature block to generate a first feature vector;

[0040] Map each first feature vector to a first anomaly score through a fully connected layer;

[0041] Select all the first anomaly scores using a \(3\times3\) convolutional layer, and select the maximum value from every 9 first anomaly scores to form a first anomaly score array, denoted as the anomaly score \(F_1\).

[0042] Optionally, the step of sending the rotation-expanded and preprocessed sample image into the invisible anomaly branch to obtain the anomaly score \(F_2\) includes:

[0043] Feature extraction is performed on the rotation-expanded and preprocessed sample images through a convolutional layer to generate a second feature map;

[0044] The first preset percentage of normal images in the rotation-expanded and preprocessed sample images are taken as standard images;

[0045] Feature extraction is performed on each standard image through a convolutional layer and the average value is taken to generate an average feature map;

[0046] The average feature map is subtracted from the second feature map to obtain a deviation feature map;

[0047] The deviation feature map is divided into multiple second feature blocks according to a preset size, and the pixel values of each second feature block are obtained to generate a second feature vector;

[0048] Each second feature vector is mapped to a second anomaly score through a fully connected layer;

[0049] A 3×3 convolutional layer is used to select all the second anomaly scores, and the maximum value is selected from every 9 second anomaly scores to form a second anomaly score array, denoted as anomaly score F2.

[0050] In a second aspect, the present invention provides a dual-branch image anomaly detection device, and the device includes:

[0051] An image acquisition module, configured to acquire an image to be detected and preprocess the image to be detected;

[0052] A score acquisition module, configured to input the preprocessed image to be detected into a trained anomaly detection model to obtain an anomaly score;

[0053] An anomaly judgment module, configured to judge whether the image to be detected is an abnormal image or a normal image according to the anomaly score;

[0054] Wherein, the training of the anomaly detection model includes:

[0055] A sample acquisition module, configured to acquire a preset number of sample images and perform rotation expansion and preprocessing on the sample images;

[0056] A model construction module, configured to construct an anomaly detection model, and the anomaly detection model includes a visible anomaly branch and an invisible anomaly branch;

[0057] A model training module, configured to train the anomaly detection model through the rotation-expanded and preprocessed sample images.

[0058] In a third aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0059] The storage medium is used to store instructions;

[0060] The processor is configured to operate according to the instructions to perform the steps of the above method.

[0061] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the above method are implemented.

[0062] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0063] A dual-branch image anomaly detection method, device, equipment and medium provided by the present invention divides anomalies into visible anomalies and invisible anomalies, calculates anomaly scores for the two types of anomalies respectively through two different branches, effectively selects features containing more information in the two branches for refinement, and obtains the final result; by training different types of industrial products, all anomalies can be taken into account simultaneously, and there is good adaptability; at the same time, a pre-trained model is used to extract features of input samples, greatly reducing the amount of calculation and significantly reducing the training parameters required for the model itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of a dual-branch image anomaly detection method provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the protection scope of the present invention.

[0066] Embodiment 1:

[0067] As Figure 1 shown, Embodiment 1 of the present invention provides a dual-branch image anomaly detection method, including the following steps:

[0068] 1. Obtain the image to be detected and preprocess the image to be detected;

[0069] In this embodiment, the preprocessing includes changing the image size to 448×448.

[0070] 2. Input the preprocessed image to be detected into the trained anomaly detection model to obtain the anomaly score.

[0071] 3. Determine whether the image to be detected is an abnormal image or a normal image according to the anomaly score; specifically:

[0072] If the anomaly score of the image to be detected is greater than or equal to 0.5 and less than or equal to 1, the image to be detected is an abnormal image;

[0073] If the anomaly score of the image to be detected is greater than or equal to 0 and less than 0.5, the image to be detected is a normal image.

[0074] Among them, the training of the anomaly detection model includes:

[0075] S1. Obtain a preset number of sample images, and perform rotation expansion and preprocessing on the sample images;

[0076] In this embodiment, the sample images can be selected from the MVTec Anomaly Detection (MVTec AD) dataset, which contains 5354 different color images in ten categories; the anomaly detection models of each category can be trained through the images of each category;

[0077] The rotation expansion of the sample images includes: rotating the sample images by 90°, 180°, and 270°, and saving the rotated sample images. Through the rotation expansion, the number of images is expanded to 4 times the original, so as to strengthen the training.

[0078] S2. Construct an anomaly detection model, which includes a visible anomaly branch and an invisible anomaly branch.

[0079] S3. Train the anomaly detection model with the sample images after rotation expansion and preprocessing.

[0080] The training process includes:

[0081] S01. Send the sample images after rotation expansion and preprocessing into the visible anomaly branch to obtain the anomaly score F1;

[0082] S02. Send the sample images after rotation expansion and preprocessing into the invisible anomaly branch to obtain the anomaly score F2;

[0083] S03. Perform global average pooling on the anomaly scores F1 and F2 respectively, and add the results of the global average pooling to obtain z 1 ;

[0084] S04. Reduce the dimension of z 1 to 1 through a fully connected layer to obtain z 2 , and calculate the attention of the visible anomaly branch and the invisible anomaly branch based on z 2 :

[0085]

[0086]

[0087] In the formula, l1 and l2 are the attention of the visible anomaly branch and the invisible anomaly branch respectively, L1 and H1 are the weight parameters of the visible anomaly branch, and L2 and H2 are the weight parameters of the invisible anomaly branch;

[0088] S05. Calculate the final anomaly score S based on the anomaly scores F1, F2 and the attention levels l1, l2:

[0089] S = norm(F1 × l1 + F2 × l2)

[0090] In the formula, norm is a [0, 1] normalization function;

[0091] S06. Calculate the model loss Loss based on the anomaly score S:

[0092] Loss = (1 - y)|D(S)| + ymax(0, a - D(S))

[0093]

[0094] In the formula, μγ, δγ are the mean and standard deviation of the obtained anomaly scores; when D(S) ∈ [0.5, 1], y = 1, when D(S) ∈ [0, 0.5), y = 0; a is a preset confidence interval parameter;

[0095] S07. Update the weight parameters L1, H1, L2, H2 according to the model loss Loss to obtain an updated anomaly detection model;

[0096] S08. Repeat the above steps (i.e., steps S01 - S07) until the model loss Loss converges or reaches the preset maximum number of iterations to obtain a trained anomaly detection model.

[0097] Where:

[0098] (1) Send the rotated and pre - processed sample images into the visible anomaly branch to obtain the anomaly score F1, including:

[0099] Extract features from the rotated and pre - processed sample images through a convolutional layer to generate a first feature map;

[0100] Divide the first feature map into multiple first feature blocks according to a preset size, and obtain the pixel values of each first feature block to generate a first feature vector;

[0101] Map each first feature vector to a first anomaly score through a fully - connected layer;

[0102] Select from all the first anomaly scores using a 3×3 convolutional layer, and select the maximum value from every 9 first anomaly scores to form a first anomaly score array, denoted as the anomaly score F1.

[0103] (2) Send the rotated and pre - processed sample images into the invisible anomaly branch to obtain the anomaly score F2, including:

[0104] Feature extraction is performed on the rotated, expanded, and preprocessed sample images through a convolutional layer to generate a second feature map;

[0105] The first preset percentage of normal images in the rotated, expanded, and preprocessed sample images are taken as standard images;

[0106] Feature extraction is performed on each standard image through a convolutional layer and the average value is taken to generate an average feature map;

[0107] The average feature map is subtracted from the second feature map to obtain a deviation feature map;

[0108] The deviation feature map is divided into multiple second feature blocks according to a preset size, and the pixel values of each second feature block are obtained to generate a second feature vector;

[0109] Each second feature vector is mapped to a second anomaly score through a fully connected layer;

[0110] A 3×3 convolutional layer is used to select all the second anomaly scores, and the maximum value is selected from every 9 second anomaly scores to form a second anomaly score array, denoted as anomaly score F2.

[0111] Embodiment 2:

[0112] An embodiment of the present invention provides a dual-branch image anomaly detection device, which includes:

[0113] An image acquisition module, configured to acquire an image to be detected and preprocess the image to be detected;

[0114] A score acquisition module, configured to input the preprocessed image to be detected into a trained anomaly detection model to obtain an anomaly score;

[0115] An anomaly judgment module, configured to judge whether the image to be detected is an abnormal image or a normal image according to the anomaly score;

[0116] Among them, the training of the anomaly detection model includes:

[0117] A sample acquisition module, configured to acquire a preset number of sample images and perform rotation expansion and preprocessing on the sample images;

[0118] A model construction module, configured to construct an anomaly detection model, and the anomaly detection model includes a visible anomaly branch and an invisible anomaly branch;

[0119] A model training module, configured to train the anomaly detection model with the rotated, expanded, and preprocessed sample images.

[0120] Embodiment 3:

[0121] Based on Embodiment 1, an embodiment of the present invention provides an electronic device, including a processor and a storage medium;

[0122] The storage medium is used to store instructions;

[0123] The processor is used to operate according to the instructions to execute the steps of the above method.

[0124] Example 4:

[0125] Based on Example 1, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0128] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocksFigure 1 Steps of functions specified in one or more boxes.

[0130] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A dual-branch image anomaly detection method, characterized in that, Including: Obtain the image to be detected and preprocess the image to be detected; Input the preprocessed image to be detected into the trained anomaly detection model to obtain an anomaly score; Judge whether the image to be detected is an abnormal image or a normal image according to the anomaly score; Among them, the training of the anomaly detection model includes: Obtain a preset number of sample images and perform rotation expansion and preprocessing on the sample images; Construct an anomaly detection model, and the anomaly detection model includes a visible anomaly branch and an invisible anomaly branch; Train the anomaly detection model with the sample images after rotation expansion and preprocessing, and the training steps include: Send the sample images after rotation expansion and preprocessing into the visible anomaly branch to obtain an anomaly score F1; Send the sample images after rotation expansion and preprocessing into the invisible anomaly branch to obtain an anomaly score F2; Perform global average pooling on the anomaly scores F1 and F2 respectively, and add the results of the global average pooling to obtain z 1 ; Reduce the dimension of z to 1 through a fully connected layer 1 to obtain z 2 , and calculate the attention of the visible anomaly branch and the invisible anomaly branch based on z 2 : In the formula, l1 and l2 are the attentions of the visible anomaly branch and the invisible anomaly branch respectively, L1 and H1 are the weight parameters of the visible anomaly branch, and L2 and H2 are the weight parameters of the invisible anomaly branch; Calculate the final anomaly score S according to the anomaly scores F1, F2 and the attentions l1, l2: S = norm(F1×l1 + F2×l2) In the formula, norm is the [0,1] normalization function; Calculate the model loss Loss based on the anomaly score S: Loss = (1 - y)|D(S)| + ymax(0, a - D(S)) In the formula, μγ and δγ are the mean and standard deviation of the obtained anomaly scores; when D(S) ∈ [0.5, 1], y = 1, when D(S) ∈ [0, 0.5), y = 0; a is the preset confidence interval parameter; Update the weight parameters L1, H1, L2, H2 according to the model loss Loss to obtain the updated anomaly detection model; Repeat the above training steps until the model loss Loss converges or reaches the preset maximum number of iterations to obtain the trained anomaly detection model.

2. The double-branch image anomaly detection method according to claim 1, wherein The preprocessing includes changing the image size to 448×448.

3. The dual-branch image anomaly detection method according to claim 1, wherein The judging whether the image to be detected is an abnormal image or a normal image according to the anomaly score includes: If the anomaly score of the image to be detected is greater than or equal to 0.5 and less than or equal to 1, the image to be detected is an abnormal image; If the anomaly score of the image to be detected is greater than or equal to 0 and less than 0.5, the image to be detected is a normal image.

4. The dual-branch image anomaly detection method according to claim 1, wherein The rotation expansion of the sample images includes: Rotate the sample images by 90°, 180°, 270°, and save each rotated sample image.

5. The dual-branch image anomaly detection method according to claim 1, wherein The sending the sample images after rotation expansion and preprocessing into the visible anomaly branch to obtain an anomaly score F1 includes: Extract features from the sample images after rotation expansion and preprocessing through a convolutional layer to generate a first feature map; Divide the first feature map into multiple first feature blocks according to a preset size, and obtain the pixel values of each first feature block to generate a first feature vector; Map each first feature vector to a first anomaly score through a fully connected layer; Select all the first anomaly scores by using a 3×3 convolutional layer, and select the maximum value from every 9 first anomaly scores to form a first anomaly score array, denoted as the anomaly score F1.

6. The double-branch image anomaly detection method according to claim 5, characterized in that, Feeding the rotation-expanded and preprocessed sample images into the invisible anomaly branch to obtain the anomaly score F2 includes: Performing feature extraction on the rotation-expanded and preprocessed sample images through a convolutional layer to generate a second feature map; Taking the first preset percentage of normal images in the rotation-expanded and preprocessed sample images as standard images; Performing feature extraction on each standard image through a convolutional layer and taking the average value to generate an average feature map; Subtracting the average feature map from the second feature map to obtain a deviation feature map; Dividing the deviation feature map into multiple second feature blocks according to a preset size, and obtaining the pixel values of each second feature block to generate a second feature vector; Mapping each second feature vector to a second anomaly score through a fully connected layer; Using a 3×3 convolutional layer to select all the second anomaly scores, and selecting the maximum value from every 9 second anomaly scores to form a second anomaly score array, denoted as the anomaly score F2.

7. A dual-branch image anomaly detection device, characterized in that, The device includes: An image acquisition module, configured to acquire a to-be-detected image and preprocess the to-be-detected image; A score acquisition module, configured to input the preprocessed to-be-detected image into a trained anomaly detection model to obtain an anomaly score; An anomaly determination module, configured to determine whether the to-be-detected image is an abnormal image or a normal image according to the anomaly score; Wherein, the training of the anomaly detection model includes: A sample acquisition module, configured to acquire a preset number of sample images and perform rotation expansion and preprocessing on the sample images; A model construction module, configured to construct an anomaly detection model, and the anomaly detection model includes a visible anomaly branch and an invisible anomaly branch; A model training module, configured to train the anomaly detection model through the rotation-expanded and preprocessed sample images, and the training steps include: Feeding the rotation-expanded and preprocessed sample images into the visible anomaly branch to obtain the anomaly score F1; Feeding the rotation-expanded and preprocessed sample images into the invisible anomaly branch to obtain the anomaly score F2; Perform global average pooling on the anomaly scores F1 and F2 respectively, and add the results of global average pooling to obtain z 1 ; Reduce the dimension of z to 1 through the fully connected layer 1 to obtain z 2 , and calculate the attention of the visible anomaly branch and the invisible anomaly branch based on z 2 : In the formula, l1 and l2 are the attentions of the visible anomaly branch and the invisible anomaly branch respectively, L1 and H1 are the weight parameters of the visible anomaly branch, and L2 and H2 are the weight parameters of the invisible anomaly branch; Calculating the final anomaly score S according to the anomaly scores F1, F2 and the attentions l1, l2: S = norm(F1×l1 + F2×l2) In the formula, norm is a [0,1] normalization function; Calculating the model loss Loss based on the anomaly score S: Loss = (1 - y)|D(S)| + ymax(0, a - D(S)) In the formula, μγ and δγ are the mean and standard deviation of the obtained anomaly scores; when D(S) ∈ [0.5, 1], y = 1, when D(S) ∈ [0, 0.5), y = 0; a is a preset confidence interval parameter; Updating the weight parameters L1, H1, L2, H2 according to the model loss Loss to obtain an updated anomaly detection model; Repeating the above training steps until the model loss Loss converges or reaches a preset maximum number of iterations to obtain a trained anomaly detection model.

8. An electronic device, characterized in that, Including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instruction to perform the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

Images