An image anomaly detection method based on single-expert double-apprentice

Abstract

The present application belongs to the technical field of image anomaly detection, and discloses an image anomaly detection method based on single-expert dual-apprentice, which comprises an expert network, two apprentice networks, and is divided into two stages of training and testing. In the training stage, the two apprentice networks are subjected to similarity learning using unknown abnormal images and real known abnormal images respectively, so that the normal features between the two apprentice networks and the expert network have high similarity, and the features of real known and unknown abnormal data have low similarity. In the testing stage, the three networks are used respectively to extract features of the test images, the similarity between the two apprentice networks and the expert network is calculated, and the negative value is taken as the abnormal score. Then, the abnormal scores generated by the two apprentice networks are added to obtain the final abnormal score. The method effectively utilizes the small amount of abnormal information in industrial products and improves the performance of automatic quality inspection.

Description

Technical Field

[0001] This invention belongs to the technical field of image anomaly detection, and more specifically, relates to an image anomaly detection method based on a single expert and dual apprenticeship. Background Technology

[0002] In industrial manufacturing, various unpredictable factors such as machine malfunctions and worker errors often lead to quality problems in industrial products, typically manifesting as surface defects. To improve the yield rate of delivered products, quality inspection departments are commonly established in industrial production. Product quality inspection methods are gradually shifting from traditional worker-based inspections to automated inspection methods based on machine vision, with anomaly detection being a prime example. Due to the limited number of anomalous samples in actual production, most existing anomaly detection methods are unsupervised, meaning that only the distribution of normal samples is known during model training. However, a small number of anomalous samples still contains a valid description of the anomalous distribution, and using a small number of anomalous samples during training can help further improve the performance of the anomaly detection model. Since the types of anomalous samples are diverse and unknown, a small number of anomalous samples cannot fully describe the distribution of all anomalous samples. Therefore, the anomaly detection problem using a small number of anomalous samples is essentially an open-set supervised anomaly detection problem, meaning that normal samples and a small number of anomalous samples are used during training, while the unknown anomalous samples need to be identified during the testing phase. Therefore, how to identify unknown anomalous samples based on a small number of anomalous samples is of great significance. Summary of the Invention

[0003] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an image anomaly detection method based on a single expert and dual apprenticeship. This method can detect both unknown and known anomalies in image anomalies using normal samples and a small number of abnormal samples, thereby improving the ability to detect image anomalies.

[0004] To achieve the above objectives, according to one aspect of the present invention, an image anomaly detection method based on a single expert and two apprentices is provided. The method includes: S1: acquiring a trained feature extraction deep neural network, i.e., an expert network, and two untrained feature extraction deep neural networks, i.e., a first apprentice network and a second apprentice network; the training process of the first apprentice network includes: S2: randomly generating an abnormal shape to obtain an abnormal image mask, and synthesizing the abnormal image mask with a normal image and a natural texture image of the normal image to obtain an abnormal image; S3: inputting the abnormal image into the expert network and the first apprentice network to obtain a first expert feature and a first apprentice feature, respectively; S4: inputting the first expert feature, the first apprentice feature, and the abnormal image mask into a first difference learner to constrain the cosine similarity between the first expert feature, the first apprentice feature, and the abnormal image mask to obtain a first difference result, and performing backpropagation based on the first difference result to achieve image anomaly detection. The parameter optimization of the first apprentice network and the training process of the second apprentice network are described below: S2': Randomly collect the same number of images as the real abnormal images from normal images to obtain normal images, real abnormal images, and a mask of real abnormal images; S3': Input the real abnormal images into the expert network and the second apprentice network to obtain second expert features and second apprentice features. Input the second expert features, the second apprentice features, and the mask of real abnormal images into the second difference learner to obtain the second difference result. Perform backpropagation based on the second difference result to optimize the parameters of the second apprentice network; S5: Input the image to be tested into the expert network, the parameter-optimized first apprentice network, and the second apprentice network to obtain expert features, first features, and second features respectively. Calculate the first similarity between the expert features and the first features, and the second similarity between the expert features and the second features. Add the first similarity and the second similarity to obtain the abnormal region of the image to be tested.

[0005] Preferably, step S4, which involves constraining the cosine similarity between the first expert feature, the first apprentice feature, and the abnormal image mask to obtain the first difference result, specifically includes: S41: Obtaining the normal feature F extracted by the expert network based on the abnormal image mask. ε N Normal features extracted from the first apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the first apprentice network S42: Construct a loss function for training the first apprentice network, and use the loss function to constrain the cosine similarity, wherein the loss function of the first apprentice network ξ1 is:

[0006]

[0007] Where, N N for and The sum of the quantities, N A for and The quantity and.

[0008] Preferably, step S41 specifically involves: obtaining the pixel distribution of the abnormal image mask; and based on the pixel distribution, finding the corresponding pixels in the first expert feature and the first apprentice feature to obtain the normal features extracted by the expert network. Normal features extracted by the first apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the first apprentice network

[0009] Preferably, step S2 specifically includes: obtaining an abnormal image mask M by randomly generating an abnormal shape. S Using the aforementioned abnormal image mask M S For normal image I n Natural texture image I compared to normal image t Synthesizing to obtain an abnormal image I S The synthesis formula is as follows:

[0010]

[0011] in, For M S The mask after inversion.

[0012] Preferably, step S3', which involves inputting the second expert feature, the second apprentice feature, and the mask corresponding to the real abnormal image into the second difference learner to obtain the second difference result, specifically includes: S31': obtaining the normal features extracted by the expert network based on the mask of the real abnormal image. Normal features extracted by the second apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the second apprentice network S32': Construct the loss function for training the second apprentice network, and use the loss function to constrain the cosine similarity, wherein the loss function of the second apprentice network ξ2 is:

[0013]

[0014] Where, N N for and The sum of the quantities, NA for and The quantity and.

[0015] Preferably, step S31' specifically involves: obtaining the pixel distribution of the real abnormal image mask; and based on the pixel distribution, finding the corresponding pixels in the second expert feature and the second apprentice feature to obtain the normal features extracted by the expert network. Normal features extracted by the second apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the second apprentice network

[0016] Preferably, in step S5, the first similarity between the expert feature and the first feature is calculated. The calculation formula is:

[0017]

[0018] Calculate the second similarity between the expert feature and the second feature. The calculation formula is:

[0019]

[0020] Where i and j are the pixel coordinates at the corresponding pixel positions.

[0021] Preferably, in step S5, adding the first similarity and the second similarity to obtain the abnormal region of the image under test specifically involves adding the corresponding pixels of the first similarity and the second similarity to obtain the abnormality score A of the image under test. f The abnormal region is determined based on the abnormal score, using the following formula:

[0022]

[0023] In summary, compared with the prior art, the image anomaly detection method based on a single expert and dual apprenticeship provided by this invention has the following beneficial effects:

[0024] 1. This application achieves the detection capability of the first apprentice network for abnormal images by randomly constructing abnormal images and comparing the results obtained by the trained image extraction expert network and the untrained first apprentice network with the abnormal images. The second apprentice network achieves the detection capability of the real abnormal images by comparing the features of the real abnormal images and the expert network. In this way, the detection capability of both real abnormal images and unknown abnormalities is achieved, thus improving the detection capability.

[0025] 2. By applying cosine constraints to the prediction results, the similarity between the prediction results of the first apprentice network, the second apprentice network, and the expert network in the prediction results of the normal images is increased, while the similarity between the prediction results of the first apprentice network, the second apprentice network, and the expert network in the prediction results of the expert network is decreased. This makes the similarity between abnormal images significantly lower than that between normal data, and thus the degree of abnormality of the image can be seen from the degree of similarity.

[0026] 3. By constructing an anomalous image mask, unknown anomalous images can be synthesized, thereby enabling the detection of unknown anomalous images.

[0027] 4. The abnormality score can be used to intuitively determine the abnormal areas and the degree of abnormality in an image. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the steps of the image anomaly detection method based on a single expert and dual apprenticeship in this application.

[0029] Figure 2 This is a flowchart of an image anomaly detection method based on a single expert and dual apprenticeship, as described in this application.

[0030] Figure 3 This is a schematic diagram illustrating the generation of abnormal images in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the difference learner in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram illustrating the image anomaly detection method based on a single expert and dual apprentices according to an embodiment of this application for detecting the image under test. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Please see Figure 1 and Figure 2 This invention proposes an image anomaly detection method based on a single expert and two apprentices, which includes the following steps S1 to S5, and the specific steps are as follows.

[0035] S1: Obtain the trained feature extraction deep neural network, namely the expert network, and two untrained feature extraction deep neural networks, namely the first apprentice network and the second apprentice network.

[0036] The expert network is a pre-trained deep neural network for feature extraction. Any existing feature extraction network with strong judgment capabilities can be used, such as ResNet, VggNet, and HRNet, which are based on deep convolutional network architectures. The first and second apprentice networks can have the same or different architectures as the expert network. The difference is that the first and second apprentice networks are not trained.

[0037] The first apprentice network is trained using unknown anomalous images, while the second apprentice network is trained using real anomalous images. The training process of the first apprentice network is as follows: steps S2 to S4. The training process of the second apprentice network is as follows: steps S2' and S3'. There is no strict restriction on the order of training the first and second apprentice networks. The specific process is as follows.

[0038] The training process of the first apprentice network:

[0039] S2: Randomly generate abnormal shapes to obtain abnormal image masks, and synthesize the abnormal image mask with normal images and natural texture images of normal images to obtain abnormal images.

[0040] An abnormal image mask can be obtained by randomly generating abnormal shapes, for example, by using professional software such as Photoshop.

[0041] like Figure 3 As shown, the abnormal image mask M is used. S For normal image I n Natural texture image I compared to normal image t Synthesizing to obtain an abnormal image I S The synthesis formula is as follows:

[0042]

[0043] in, For M S The mask after inversion.

[0044] S3: Input the abnormal image into the expert network and the first apprentice network to obtain the first expert feature and the first apprentice feature, respectively.

[0045] Abnormal image I S The first expert feature F is obtained by inputting the expert network ε. ε , will abnormal image I S Input the first apprentice network ξ1 to obtain the first apprentice features

[0046] S4: Input the first expert feature, the first apprentice feature, and the abnormal image mask into the first difference learner to constrain the cosine similarity between the first expert feature, the first apprentice feature, and the abnormal image mask to obtain the first difference result. Perform backpropagation based on the first difference result to optimize the parameters of the first apprentice network.

[0047] like Figure 4 As shown, the first expert feature F ε Characteristics of the first apprentice and the abnormal image mask M S Inputting a first difference learner D1, the first difference learner D1 can process the first expert feature F ε Characteristics of the first apprentice and the abnormal image mask M S The first difference result is obtained by constraining the cosine similarity between them, as detailed below:

[0048] S41: Based on the abnormal image mask M S Obtain normal features extracted by expert networks Normal features extracted by the first apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the first apprentice network Specifically as follows:

[0049] S411: Obtain the pixel distribution of the abnormal image mask;

[0050] S412: Based on the pixel distribution, find the corresponding pixel in the first expert feature and the first apprentice feature respectively, and then obtain the normal feature extracted by the expert network. Normal features extracted by the first apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the first apprentice network

[0051] S42: Construct a loss function for training the first apprentice network, and use the loss function to constrain the cosine similarity, wherein the loss function of the first apprentice network ξ1 is:

[0052]

[0053] Where, N N for and The sum of the quantities, N A for and The quantity and.

[0054] By following the steps above, the similarity of normal features can be maximized and the similarity of abnormal features can be minimized.

[0055] Backpropagation is performed based on the first difference result to optimize the parameters of the first apprentice network.

[0056] The training process of the second apprentice network:

[0057] S2': Randomly collect the same number of images as the real abnormal images from the normal images to obtain the normal images, the real abnormal images, and the real abnormal image mask.

[0058] Because the amount of real abnormal data is small and the number of real abnormal data and normal data is extremely imbalanced, this application needs to use sampling to ensure that the probability of real abnormal data and normal data appearing during training is equal. The preferred sampling method is a weighted random sampling method.

[0059] S3': Input the real anomaly image into the expert network and the second apprentice network to obtain the second expert feature and the second apprentice feature. Input the second expert feature, the second apprentice feature, and the real anomaly image mask into the second difference learner to obtain the second difference result. Perform backpropagation based on the second difference result to optimize the parameters of the second apprentice network.

[0060] The second difference learner D2 has the same function as the first difference learner D1: to constrain the cosine similarity of the results.

[0061] Step S3' involves inputting the second expert feature, the second apprentice feature, and the mask corresponding to the real anomaly image into the second difference learner to obtain the second difference result. Specifically, this includes:

[0062] S31': Obtain normal features extracted by the expert network based on a mask of a real abnormal image. Normal features extracted by the second apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the second apprentice network

[0063] Step S31' specifically includes:

[0064] S311': Obtain the pixel distribution of the real abnormal image mask;

[0065] S312': Based on the pixel distribution, find the corresponding pixel in the second expert feature and the second apprentice feature respectively, and then obtain the normal feature extracted by the expert network. Normal features extracted by the second apprentice network Anomaly features extracted by expert networks and the abnormal features extracted from the second apprentice network

[0066] S32': Construct the loss function for training the second apprentice network, and use the loss function to constrain the cosine similarity, wherein the loss function of the second apprentice network ξ2 is:

[0067]

[0068] Where, N N for and The sum of the quantities, N A for and The quantity and.

[0069] S5: Input the image to be tested into the expert network, the first apprentice network and the second apprentice network after parameter optimization to obtain expert features, first features and second features respectively. Calculate the first similarity between the expert features and the first features, and the second similarity between the expert features and the second features. Add the first similarity and the second similarity to obtain the abnormal region of the image to be tested.

[0070] like Figure 5 As shown, when testing an image, the image to be tested needs to be simultaneously input into the expert network ε, the first apprentice network ξ1 with optimized parameters, and the second apprentice network ξ2 to obtain expert features. First characteristic Second feature

[0071] First similarity The calculation formula is:

[0072]

[0073] Second similarity The calculation formula is:

[0074]

[0075] Where i and j are the pixel coordinates at the corresponding pixel positions.

[0076] In step S5, the abnormal region of the image under test is obtained by adding the first similarity and the second similarity. Specifically, the abnormality score A of the image under test is obtained by adding the corresponding pixels of the first similarity and the second similarity. f The abnormal region is determined based on the abnormal score, using the following formula:

[0077]

[0078] Furthermore, abnormal regions can be identified based on anomaly scores, and the degree of abnormality within these regions can be determined accordingly. This enables the detection of both unknown and known abnormal images, thereby improving detection capabilities. This further enhances the automated inspection capabilities of industrial quality control.

[0079] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A single-expert dual-apprentice based image anomaly detection method, characterized in that, The method comprises: S1: obtaining a trained feature extraction deep neural network, i.e., an expert network, and two untrained feature extraction deep neural networks, i.e., a first apprentice network and a second apprentice network; The training process of the first apprentice network comprises: S2: randomly generating an abnormal shape to obtain an abnormal image mask, synthesizing the abnormal image mask with a normal image and a natural texture image of the normal image to obtain an abnormal image; S3: inputting the abnormal image into the expert network and the first apprentice network to obtain first expert features and first apprentice features, respectively; S4: inputting the first expert features, the first apprentice features, and the abnormal image mask into a first difference learner to constrain the cosine similarity between the first expert features, the first apprentice features, and the abnormal image mask, and then obtain a first difference result, and performing back propagation according to the first difference result to optimize the parameters of the first apprentice network; The training process of the second apprentice network comprises: S2': randomly collecting the same number of images as the number of real abnormal images in the normal images to obtain normal images, real abnormal images, and real abnormal image masks; S3': inputting the real abnormal images into the expert network and the second apprentice network to obtain second expert features and second apprentice features, and inputting the second expert features, the second apprentice features, and the real abnormal image masks into a second difference learner to obtain a second difference result, and performing back propagation according to the second difference result to optimize the parameters of the second apprentice network; S5: inputting a to-be-tested image into the expert network, the first apprentice network, and the second apprentice network to obtain expert features, first features, and second features, respectively, calculating a first similarity between the expert features and the first features, and a second similarity between the expert features and the second features, and adding the first similarity and the second similarity to obtain an abnormal region of the to-be-tested image.

2. The method of claim 1, wherein, The step S4 specifically comprises: S41: obtaining normal features extracted by the expert network according to the abnormal image mask normal features extracted by the first apprentice network abnormal features extracted by the expert network and abnormal features extracted by the first apprentice network S42: constructing a loss function for training the first apprentice network, and using the loss function to constrain the cosine similarity, wherein the loss function of the first apprentice network ξ1 is: where N N is the number of and N is the number of A and N is the number of .

3. The method of claim 2, wherein, The step S41 specifically comprises: Obtaining the pixel distribution of the abnormal image mask; According to the pixel distribution, corresponding pixel points in the first expert feature and the first apprentice feature are found respectively to obtain normal features extracted by the expert network Normal features extracted by the first apprentice network Abnormal features extracted by the expert network and abnormal features extracted by the first apprentice network 4. The method of claim 1, wherein, The step S2 specifically comprises: An abnormal image mask M is obtained by a randomly generated abnormal shape S An abnormal image I S is synthesized from a normal image I n and a natural texture image I t of the normal image S , and the synthesis formula is as follows: wherein M is S Mask after negation.

5. The method of claim 1, wherein, The step S3' specifically comprises: S31’: normal features extracted by the expert network according to the real abnormal image mask normal features extracted by the second apprentice network abnormal features extracted by the expert network and abnormal features extracted by the second apprentice network S32': constructing a loss function for training the second apprentice network, and using the loss function to constrain the cosine similarity, wherein the loss function of the second apprentice network ξ2 is: where N N is the number of and N A is the number of and ​ 6. The method of claim 5, wherein, The step S31' specifically comprises: Obtaining the pixel distribution of the real abnormal image mask; According to the pixel distribution, corresponding pixel points in the second expert feature and the second apprentice feature are found respectively to obtain normal features extracted by the expert network Normal features extracted by the second apprentice network Abnormal features extracted by the expert network and abnormal features extracted by the second apprentice network 7. The method of claim 1, wherein, calculating a first similarity of the expert feature and the first feature in step S5 The calculation formula is: computing a second similarity of the expert feature to the second feature The formula for computing the second similarity is: Wherein, i and j are pixel coordinates at corresponding pixel positions.

8. The method of claim 7, wherein, The first similarity and the second similarity are added in step S5 to obtain the abnormal region of the image to be tested, specifically, corresponding pixel points of the first similarity and the second similarity are added to obtain an abnormal score A of the image to be tested f The abnormal region is obtained according to the abnormal score, and the specific formula is as follows:

Images