An image anomaly detection method considering label uncertainty

Abstract

The present invention belongs to the technical field related to image anomaly detection, and discloses an image anomaly detection method that takes label uncertainty into account. The method includes an expert network and an apprentice network, and is divided into two stages: training and testing. In the training stage, a spatial uncertainty mask is generated based on the training image mask. The training image is then input into the expert network and the apprentice network to extract expert features and apprentice features, and all features are divided into confusing features, normal features, and abnormal features based on the training image mask and the spatial uncertainty mask. During the optimization process, the similarity of normal features is increased and the similarity of abnormal features is reduced. The confusing features are not explicitly optimized to avoid the problem of optimization direction ambiguity. In the testing stage, the expert network and the apprentice network are used to extract features from the test image, and the similarity between the apprentice network and the expert network is calculated. The negative of the similarity is the abnormal area. This method takes into account the label uncertainty in industrial product images and improves the image anomaly detection capability.

Description

Technical Field

[0001] The present invention belongs to the technical field related to image anomaly detection, and more specifically, relates to an image anomaly detection method considering label uncertainty. Background Art

[0002] In recent years, intelligent industrial manufacturing has become a growing trend, and a key component of this intelligence is intelligent industrial quality inspection. During actual industrial production, various unforeseen circumstances, such as machine failures and worker misoperation, can lead to quality issues in industrial products, such as surface defects (also known as surface quality anomalies). Existing technologies often rely on manual visual inspection, but worker capabilities vary, making it difficult to guarantee product yields. Automated industrial quality inspection technology based on machine vision promises to significantly improve the level of intelligent industrial quality inspection.

[0003] Existing intelligent machine vision quality inspection algorithms are usually image anomaly detection algorithms based on deep learning, which require a large amount of pixel-level labeled data. However, the cost of pixel-level labeling of images is high, and different workers use different labeling standards, resulting in uncertainty in the labels obtained. Specifically, the uncertainty of industrial image labels is manifested in the following aspects: different workers have different judgments on the location of defects in the image, resulting in three situations in the labeling process: under-labeling, fine labeling, and over-labeling, such as Figure 1 As shown in Figure 2, even if the annotations are very detailed, the labeled boundary areas are prone to ambiguity. Therefore, considering label uncertainty is expected to improve the performance of existing image anomaly detection. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides an image anomaly detection method that takes label uncertainty into account. By eliminating confusing features and then not explicitly optimizing the confusing features, ambiguity in the optimization direction is avoided. During the optimization process, the similarity of normal features is increased and the similarity of abnormal features is reduced, thereby improving the image anomaly detection capability.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present invention, a method for image anomaly detection considering label uncertainty is provided, the method comprising: S1: obtaining a training image mask based on a training image in a training image data set, dilating the training image mask and subtracting the mask pixel by pixel from the training image mask to obtain a spatial uncertainty mask; S2: obtaining a trained feature extraction deep neural network, i.e., an expert network, and an untrained feature extraction deep neural network, i.e., an apprentice network; S3: inputting the training image into the expert network and the apprentice network respectively to obtain expert features and apprentice features; S4: according to the pixel values ​​at each pixel position of the training image mask and the spatial uncertainty mask, The expert features and apprentice features are further divided into expert confusion features, expert normal features, expert abnormal features, apprentice confusion features, apprentice normal features and apprentice abnormal features; S5: constrain the cosine similarity between the expert normal features and the apprentice normal features, as well as the cosine similarity between the expert abnormal features and the apprentice abnormal features, to obtain constraint results, and perform backpropagation based on the constraint results to optimize the parameters of the apprentice network; S6: input the image to be tested into the expert network and the apprentice network with optimized parameters at the same time to obtain expert extracted features and apprentice extracted features, calculate the similarity value between the expert extracted features and the apprentice extracted features, and obtain the abnormal area of ​​the image to be tested according to the similarity value.

[0006] Preferably, step S4 is specifically as follows: the expert features and apprentice features corresponding to the pixel coordinates that are 1 in the spatial uncertainty mask are discriminated as expert confusion features and apprentice confusion features; the expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 0 in the training image mask are discriminated as expert normal features and apprentice normal features; the expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 1 in the training image mask are discriminated as expert abnormal features and apprentice abnormal features.

[0007] Preferably, the constraint function L for constraining the cosine similarity between the normal features of the expert and the normal features of the apprentice, and the cosine similarity between the abnormal features of the expert and the abnormal features of the apprentice in step S5 is:

[0008]

[0009] in, Normal characteristics for experts, Normal characteristics for apprentices, For expert abnormal characteristics, Apprentice abnormal characteristics, N N for and The number and N A for and The number and.

[0010] Preferably, step S1 specifically includes: constructing a morphological operation kernel 1 w×w , where w is a preset parameter; using the morphological operation kernel 1 w×w Perform an expansion operation on the training image mask M; subtract the expanded mask from the original training image mask M pixel by pixel to obtain the spatial uncertainty mask M u , the specific formula is as follows:

[0011]

[0012] in, For the expansion operation, It is a pixel-level subtraction operation.

[0013] Preferably, in step S6, the similarity value A between the features extracted by the expert and the features extracted by the apprentice is calculated using the following formula:

[0014]

[0015] Among them, i and j are the pixel coordinates at the corresponding pixel position, F ε Extract features for experts, F ξ Extract features for apprentices.

[0016] Preferably, in step S6, when the similarity value is a negative value, the corresponding area is an abnormal area.

[0017] Preferably, step S4 further includes removing the expert confusion features and apprentice confusion features.

[0018] In general, compared with the prior art, the above technical solution conceived by the present invention provides an image anomaly detection method that considers label uncertainty, which has the following beneficial effects:

[0019] 1. This application constructs a spatial uncertainty mask to eliminate confusing features, and avoids optimization direction ambiguity by not explicitly optimizing the confusing features, thereby avoiding label uncertainty. In the training process of the constructed network, the similarity of normal features is improved and the similarity of abnormal features is reduced, making image detection more accurate.

[0020] 2. By constructing a constraint function, the similarity of normal features between the apprentice network and the expert network is increased, while the similarity of abnormal features is reduced. Therefore, the similarity of the output features of the apprentice network and the expert network can effectively represent the degree of abnormality at the pixel level of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a conventional annotation schematic diagram in the prior art;

[0022] Figure 2 This is a schematic diagram of feature confusion caused by labels;

[0023] Figure 3 This is a step diagram of the image anomaly detection method considering label uncertainty in this application;

[0024] Figure 4 is a flow chart of the image anomaly detection method considering label uncertainty in this application;

[0025] Figure 5 is a schematic diagram of spatial uncertainty mask generation;

[0026] Figure 6 It is a schematic diagram of the optimization results before and after considering label uncertainty;

[0027] Figure 7 This is a schematic diagram of the testing phase of the image anomaly detection method of this application considering label uncertainty. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0029] Due to label uncertainty during training image detection, the normal data features and abnormal data features extracted by the expert network may have great similarities. For example, there are both normal features and abnormal features around the boundary of the labeled area, such as Figure 2As shown, it is called confusion feature. In the traditional knowledge distillation scheme, the apprentice network is expected to output normal features with high similarity to the expert network and abnormal features with low similarity, and the existence of confusion features leads to ambiguity in the optimization direction of the apprentice network. In order to avoid the optimization ambiguity caused by confusion features, this application considers label uncertainty when optimizing the apprentice network. Specifically, this application first generates a spatial uncertainty mask based on the training image mask. Then, the training image is input into the expert network and the apprentice network respectively to obtain expert features and apprentice features. Furthermore, the training image mask and the spatial uncertainty mask are used to classify the obtained features to obtain confusion features, normal features, and abnormal features in the expert network and the apprentice network. The optimization of confusion features will lead to ambiguity in the optimization direction. Therefore, this application only optimizes normal features and abnormal features. Specifically, by optimizing the apprentice network parameters, the similarity of normal features of the apprentice network and the expert network is increased, and the similarity of abnormal features is reduced. At this point, the similarity of the output features between the apprentice network and the expert network can effectively characterize the normality of the image pixels. As shown Figure 3 and Figure 4 As shown, the specific method is as follows: steps S1 to S6.

[0030] S1: A training image mask is obtained according to a training image in a training image dataset, and the training image mask is expanded and then subtracted pixel by pixel from the training image mask to obtain a spatial uncertainty mask.

[0031] like Figure 5 As shown, a training image mask M of the training image can be obtained by using image editing software such as Photoshop, and the training image mask is expanded. The expanded mask is subtracted pixel by pixel from the training image mask to obtain a spatial uncertainty mask M. u The specific steps are as follows:

[0032] S11: Constructing morphological operation core 1 w×w , where w is a preset parameter; the larger w is, the larger the generated spatial uncertainty mask is, that is, the more features are attributed to confusing features.

[0033] S12: Use the morphological operation kernel 1 w×w Perform dilation operation on the training image mask M;

[0034] S13: Subtract the expanded mask from the original training image mask M pixel by pixel to obtain the spatial uncertainty mask M u , the obtained spatial uncertainty mask is essentially the extension of the label boundary part, and the specific formula is as follows:

[0035]

[0036] in, For the expansion operation, It is a pixel-level subtraction operation.

[0037] S2: Obtain the trained feature extraction deep neural network, i.e., the expert network ε, and the untrained feature extraction deep neural network, i.e., the apprentice network ξ.

[0038] The expert network ε is a trained deep neural network for feature extraction. Any existing feature extraction network with strong judgment capabilities can be used, such as ResNet, VggNet, and HRNet networks based on deep convolutional network architectures. The apprentice network ξ can have the same or different architecture as the expert network ε, except that it is untrained.

[0039] S3: Input the training image into the expert network and the apprentice network respectively to obtain expert features and apprentice features.

[0040] Input the training image I into the expert network ε and the apprentice network ξ to obtain the expert features F ε and Apprentice Characteristics F ξ .

[0041] F ε =ε(I)

[0042] F ξ =ξ(I)

[0043] If we directly use the expert feature F ε and Apprentice Characteristics F ξ When optimizing the apprentice network, the existence of confusing features will lead to ambiguity in the optimization direction, so the confusing features need to be removed.

[0044] S4: Based on the pixel values ​​at each pixel position of the training image mask and the spatial uncertainty mask, the expert features and the apprentice features are further divided into expert confusion features, expert normal features, expert abnormal features, apprentice confusion features, apprentice normal features and apprentice abnormal features. The expert features and apprentice features are grouped based on the training image mask and the spatial uncertainty mask, specifically:

[0045] The expert features and apprentice features corresponding to the pixel coordinates of 1 in the spatial uncertainty mask are identified as expert confusion features Confusing features with Apprentice

[0046] The expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 0 in the training image mask are identified as expert normal features and normal characteristics of apprentices

[0047] The expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 1 in the training image mask are identified as expert anomaly features and Apprentice Abnormal Characteristics

[0048] Eliminate expert confusion features from expert features Confusion of characteristics with the apprentice in the apprentice characteristics

[0049] S5: Constrain the cosine similarity between the normal features of the expert and the normal features of the apprentice, as well as the cosine similarity between the abnormal features of the expert and the abnormal features of the apprentice, such as Figure 6 As shown, a constraint result is obtained, and back propagation is performed based on the constraint result to optimize the parameters of the apprentice network.

[0050] The constraint function L for constraining the cosine similarity between the normal features of the expert and the normal features of the apprentice, and the cosine similarity between the abnormal features of the expert and the abnormal features of the apprentice in step S5 is:

[0051]

[0052] in, Normal characteristics for experts, Normal characteristics for apprentices, For expert abnormal characteristics, Apprentice abnormal characteristics, N N for and The number and N A for and The number and.

[0053] After obtaining the constraint results, gradient backpropagation is performed to update the parameters of the apprentice network, enabling the apprentice network to output normal features with high similarity to the expert network and abnormal features with low similarity. Because the obfuscated features are not explicitly optimized, the problem of ambiguous optimization direction caused by obfuscated features is avoided. Furthermore, optimizing normal and abnormal features implicitly optimizes the similarity of the obfuscated features, achieving an overall better result.

[0054] S6: Input the image to be tested into the expert network and the apprentice network with optimized parameters at the same time to obtain expert extracted features and apprentice extracted features, calculate the similarity value of the expert extracted features and the apprentice extracted features, and obtain the abnormal area of ​​the image to be tested according to the similarity value.

[0055] like Figure 7 As shown, the image to be tested is simultaneously input into the expert network ε and the apprentice network ξ after parameter optimization to obtain the expert extraction feature Fε And apprentice extracted features F ξ .

[0056] The similarity value A between the features extracted by the expert and the features extracted by the apprentice is calculated using the following formula:

[0057]

[0058] Among them, i and j are the pixel coordinates at the corresponding pixel position, F ε Extract features for experts, F ξ Extract features for the apprentice. When the similarity value is negative, the corresponding area is an abnormal area.

[0059] After considering the spatial uncertainty of the label, the optimization direction ambiguity caused by confusing features during the optimization process can be avoided, thereby further improving the performance of automated detection in industrial quality inspection.

[0060] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image anomaly detection considering label uncertainty, characterized in that: The method comprises: S1: deriving a training image mask according to a training image in a training image dataset, dilating the training image mask and subtracting the mask pixel by pixel from the training image mask to obtain a spatial uncertainty mask; S2: Obtain the trained feature extraction deep neural network, i.e., the expert network, and the untrained feature extraction deep neural network, i.e., the apprentice network; S3: inputting the training image into the expert network and the apprentice network respectively to obtain expert features and apprentice features; S4: further dividing the expert features and the apprentice features into expert confusion features, expert normal features, expert abnormal features, apprentice confusion features, apprentice normal features, and apprentice abnormal features according to the pixel values ​​at each pixel position of the training image mask and the spatial uncertainty mask; S5: constraining the cosine similarity between the normal features of the expert and the normal features of the apprentice, and the cosine similarity between the abnormal features of the expert and the abnormal features of the apprentice, obtaining a constraint result, and performing backpropagation based on the constraint result to optimize the parameters of the apprentice network; S6: Input the image to be tested into the expert network and the apprentice network with optimized parameters at the same time to obtain expert extracted features and apprentice extracted features, calculate the similarity value of the expert extracted features and the apprentice extracted features, and obtain the abnormal area of ​​the image to be tested according to the similarity value.

2. The method according to claim 1, characterized in that Step S4 is specifically as follows: The expert features and apprentice features corresponding to the pixel coordinates of 1 in the spatial uncertainty mask are discriminated as expert confusion features and apprentice confusion features; The expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 0 in the training image mask are discriminated as the expert normal features and the apprentice normal features; The expert features and apprentice features corresponding to the pixel coordinates that are 0 in the spatial uncertainty mask and 1 in the training image mask are identified as expert anomaly features and apprentice anomaly features.

3. The method according to claim 1, characterized in that The constraint function L for constraining the cosine similarity between the normal features of the expert and the normal features of the apprentice, and the cosine similarity between the abnormal features of the expert and the abnormal features of the apprentice in step S5 is: in, Normal characteristics for experts, Normal characteristics for apprentices, For expert abnormal characteristics, Apprentice abnormal characteristics, N N for and The number and N A for and The number and.

4. The method according to claim 1, wherein Step S1 specifically includes: Constructing morphological operation core 1 w×w , where w is the preset parameter; Using the morphological operation kernel 1 w×w Perform dilation operation on the training image mask M; Subtract the expanded mask from the original training image mask M pixel by pixel to obtain the spatial uncertainty mask M u , the specific formula is as follows: in, For the expansion operation, It is a pixel-level subtraction operation.

5. The method according to claim 1, characterized in that In step S6, the similarity value A between the features extracted by the expert and the features extracted by the apprentice is calculated using the following formula: Among them, i and j are the pixel coordinates at the corresponding pixel position, F ε Extract features for experts, F ξ Extract features for apprentices.

6. The method according to claim 1 or 5, characterized in that In step S6, when the similarity value is a negative value, the corresponding area is an abnormal area.

7. The method according to claim 1, characterized in that Step S4 further includes removing the expert confusion features and the apprentice confusion features.

Images