Industrial abnormal pollution treatment method based on double-view core filtering

Through the dual-view core filtering technology, combined with global and local filtering mechanisms, potential anomalies in industrial images and supplement the normal core are eliminated, which solves the problem of poor detection results in the unlabeled data environment, and significantly improves industrial defect detection accuracy.

CN119068288AActive Publication Date: 2024-12-03CHONGQING UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411179793.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-03
Estimated Expiration
2044-08-27

Smart Images

  • Figure CN119068288A_ABST
    Figure CN119068288A_ABST
Patent Text Reader

Abstract

The invention discloses an industrial abnormal pollution treatment method based on double-view core filtering. On the basis of a small number of unlabeled abnormal samples mixed in an industrial data set, a global and local relation modeling method based on a feature core set is designed, sparse feature distribution characteristics of potential anomalies are considered, and the potential anomalies under overall and regional double view angles are eliminated. Besides, in order to supplement the deficiency of core submergence, a balanced sampling method is provided, and far points with potential abnormal / normal representation distribution are selected to be added into a core set, so that error elimination of normal data is reduced. According to the method, the limitation that the training of the industrial defect detector is easily interfered by unknown abnormal training data is specifically considered, the purification of the training environment is realized, and the anti-noise performance of the industrial defect detector is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of industrial defect detection and data management, and in particular to an industrial abnormal pollution management method based on dual-view core filtering. Background Art

[0002] The task of industrial defect detection is to detect defective industrial products and segment the defective areas to determine the degree of abnormality of each pixel in the original image. In recent years, the method of using only positive sample training for industrial defect detection technology has gradually become the mainstream, that is, training on a single category and reasoning on an invisible test set containing abnormal samples. However, there are unknown abnormal samples in the actual industrial production environment, that is, it is difficult to obtain pure and uncontaminated training data in real unlabeled scenes. Existing industrial defect detection technology is limited to ideal settings, so new solutions with more realistic and challenging interference training data are urgently needed. Summary of the invention

[0003] 1. Technical issues to be solved

[0004] In view of the shortcomings of the prior art, the present invention provides a method for controlling industrial abnormal pollution based on dual-perspective core filtering. In view of the problem that the existing industrial defect detection method based on core sets is not suitable for non-pure training environments, it is proposed how to eliminate as many unknown true abnormal samples as possible in the unlabeled industrial image training set. That is, under the premise that the unlabeled data follows the single-class hypothesis, it is studied how to use the feature space information of the samples to screen out a small number of real anomalies that are mixed in, purify the existing training sample pool as much as possible, and obtain a purer normal core set by sampling, so as to improve its defect detection effect on unknown industrial images.

[0005] (II) Technical solution

[0006] In order to solve the above problems, the present invention designs a solution for abnormal core filtering. By modeling the feature space after core sampling, the potential anomalies are double-screened and eliminated through global and local dual perspectives, and the far-point balanced sampling technology is used to make up for some missing normal cores. The specific technical solutions adopted are as follows:

[0007] A method for industrial abnormal pollution control based on dual-view core filtering, comprising the following steps:

[0008] S1: Feature extraction and core set sampling: Given an unlabeled industrial image training set X = {X1,…,X n}, where X is mixed with real abnormal images X a , where |X a|<<|X|; Following the approach of PatchCore, the corresponding feature tensors are extracted from each network layer of WideResNet, denoted by F i is the feature tensor extracted from the i-th network layer of WideResNet, F i The dimension is (b i ,l i ,l i ), b i is the number of feature channels of the i-th network layer, l i F i Regarding the feature map size of the image, F i By b i l i × i The feature maps of different sizes are composed of

[0009] For F i A single feature map is taken, and the elements at each position are traversed. The P×P neighbors of each element are sampled. With the help of edge padding operation, a new set of block-level feature maps is obtained from the single feature map, with a total of c i = l i × i indivual;

[0010] The i-th network layer of WideResNet has a dimension of (c i ,b i ,P,P)’s feature tensor F i ', obviously as i increases, the network layer deepens, c i Continuously decreasing, b i Keep increasing. The minimum value of WideResNet network layer i is i min , the maximum value is i max . i 'Along c i and b i Two dimensions are interpolated and the dimensions are converted to ( P,P), and then average pooling along the last two dimensions to obtain a dimension of The aggregate tensor F i ″; Finally, the F corresponding to each layer i ″ are concatenated and pooled, and finally the block-level embedding set Z = {Z1,…,Z m},in, m is the total number of block-level embeddings in Z, and the above feature extraction and aggregation operations are recorded as A single sample X i go through The operation can result in multiple block-level embeddings.

[0011] Based on the embedding set Z, use random linear mapping to reduce the dimension to a lower embedding space to obtain a new embedding set Z′. Then use the Core-set algorithm to sample Z′ and extract the core set to be processed. Represents the index of the core set element in Z′. In the startup phase of the Core-set algorithm, a part of the embeddings in Z′ is randomly selected, the average distance from other embeddings to the selected embeddings is calculated, and the embedding with the farthest average is taken to initialize M. In this way, Core-set is started, and sampling is iterated in a loop until the number of embeddings in M ​​reaches the expected target.

[0012] S2: Based on the core set M obtained in S1, we select potential abnormal latent sets from the perspectives of overall distribution and local association.

[0013] S3: Supplement sampling for some lost normal cores. The number of samples is the same as M a The sampling method is iteratively expanded, and each time an embedding is selected from the original embedding set Z′, which is the same as M a and the remaining cores are embedded in M\M a The furthest embedding is added to the core set, and the above steps are repeated until the predetermined number is reached to obtain the final core set M′.

[0014] S4: First, a single new test image x test Use the method described in S1 The operation extracts features and aggregates them to obtain the embedding of each block level Then calculate each block level embedding z based on M′ test The anomaly score, z test The closest distance to the embedding in M′, all block-level embeddings z test The maximum value of the anomaly score is defined as x test The anomaly score s is:

[0015]

[0016] When s is higher, it means the test sample x test The more likely it is an anomaly, the lower the size of the anomaly segmentation map can be obtained by directly using the anomaly scores of each block level compared to the image level anomaly score. In order to facilitate the comparison with the original size pixel-level annotation, the anomaly segmentation map needs to be further processed and restored to the size of the original input image. Test image x test From the input to the final abnormal segmentation map H′, we follow Figure 6 The above steps are operated, and a segmentation map H of the same size as the original input is obtained through bilinear interpolation, and then smoothed by a Gaussian filter (σ=4), and finally a smoothed version of the segmentation map H′ is obtained. Among them, the brighter the color of the pixel, the more likely it is a defect.

[0017] As a preference, the potential abnormal latent set M is selected in S2 a The process is as follows:

[0018] S2-1: Use global filters to establish global associations between the elements of M, and set thresholds to find potential anomalies that are far away from the vast majority of points and index them Record, where n represents the total number of samples in the training set X.

[0019] S2-2: Use local filters to first divide the edge points and the main cluster into the same subset. On the basis of initializing a series of small clusters, by comparing the density of each cluster, continuously merging adjacent clusters, finally obtaining several main clusters and discrete small clusters, and filtering the small clusters as a whole. The index of the selected core is recorded as

[0020] S2-3: Merge the detection results of the two filters. The two filters work independently, without any sequence or mutual coupling in input and output, and find the intersection of the candidate indexes selected by the global filter and the local filter, that is,

[0021] C f =C g ∩C l

[0022] According to C f The index in M ​​is used to extract the corresponding embedding from M, which constitutes the potential abnormal latent set M a .

[0023] Preferably, the S2-1 global filter comprises the following steps:

[0024] In order to quantify the overall sparse distribution relationship between abnormal cores and other cores, the similarity between any two core embeddings is defined based on the distance between them. Specifically, for each core embedding M i , first find out how to embed M with other cores j The Euclidean distance similarity is:

[0025]

[0026] Here, softmax is used for normalization in order to construct an asymmetric similarity matrix. Because for any two points Z′ i and They are not necessarily neighbors, so Z i With Z j The similarity between them is not equal to Z j With Z iThe similarity of needs to be compared with the distance of other points to get their respective one-way similarities. Here, D(·,·) is used to represent the similarity between the two points, and the following formula is used for unified calculation:

[0027]

[0028] Among them, ‖·‖2 represents the l2 norm, corresponding to the calculation of the Euclidean distance. ∈ represents the minimum value, which prevents the calculation error of the denominator being zero, corresponding to the distance between the two points being too close. And Z′ i The similarity with itself is set to negative infinity to prevent it from interfering with the normalization calculation.

[0029] Based on the above analysis, we can get the similarity matrix Sim of the unidirectional relationship, where each row i represents Z′ i Next, sum each column j and calculate the similarity of other points to Z′ j The total similarity of can be regarded as the overall association of other members with the core. All the accumulated results are sorted from small to large, and the embeddings in the top α% are selected and defined as global potential anomalies:

[0030]

[0031] ArgLastK(·,α) means taking the index of the last α% of members.

[0032] Preferably, S2-2 local filter comprises the following steps:

[0033] First of all, we only need to pay attention to the mutual correlation of the elements within the core set, cluster them first, and divide the embeddings into multiple small clusters. Without the help of other algorithms, the present invention follows the sampling method of PatchCore, selects the core again for the core set M obtained for the first time, and uses the preferentially selected embeddings as cluster centers to obtain a smaller core set, while other core set members are assigned to clusters corresponding to their nearest neighbors. In order to avoid some isolated discrete points being mistakenly assigned to the nearest main cluster due to too few cluster centers, it is necessary to set as many cluster centers as possible to subdivide different small clusters. To achieve this goal, sample γ% of the cluster centers of the core set, and obtain about 1 / γ% of core points per cluster on average. In the worst case, there may be only one cluster center.

[0034] The core embeddings are clustered, which is achieved here in combination with the PatchCore sampling algorithm. The core set M is further sampled using the PatchCore algorithm to obtain a smaller number of core sets, and γ% of its cores are selected. The selected core embeddings are used as the cluster centers of M. The unselected members of M are assigned to the clusters corresponding to their nearest neighbors. On average, each cluster obtains about 1 / γ% core embeddings, and then the adjacent small clusters are merged into one, so that the edge embeddings are incorporated into the main cluster. Specifically, the "cluster density" ρ is first calculated for each small cluster, which is defined as the average distance between all embeddings in the current cluster and the nearest neighbors in the cluster, that is:

[0035]

[0036] Here Q represents the index of the core embedding in the current cluster. When there is only one cluster member, ρ = 0. Then find the k nearest cluster centers according to the distance between each cluster center. Each cluster only compares the correlation with its neighbor cluster. The principle of comparison is to find the "cluster density" ρ within the two clusters. i and ρ j , if the closest distance between two clusters (i.e. the closest distance between all points in the two clusters) d is greater than ρ i and ρ j If both are small, it means that they belong to the same distribution, then the two clusters are merged and the new index set is recorded as Q′ i =Q i ∪Q j , the total number of new clusters is recorded as m′.

[0037] The above merging process only needs one round, because each cluster only needs to focus on the association with the neighboring clusters. When the cycle is over, the entire core set will build a different mesh structure. At this time, the number of embeddings of each cluster is compared with the total number of core sets. Clusters that are no more than β% are defined as local potential anomalies, and their embedding index is:

[0038]

[0039] Among them C l The corresponding embeddings in the initial core set M are local potential anomalies.

[0040] Preferably, the S3 supplementary sampling comprises the following steps:

[0041] The points to be supplemented are different from the abnormal embeddings that are removed, and at the same time, we hope to supplement the lost diverse normal core points, that is, to expand its diversity as much as possible on the basis of the remaining core set. However, these two goals actually need to be balanced with each other, and it is difficult to get the embedding that is farthest from both at the same time every time. Therefore, based on the core set Z after screening, r = {Z′ j |j∈C\Cf} and the potential abnormal core set Z that is screened out f = {Z′ j |j∈C f}, from the remaining embedding Z fr = {Z′ i |i∈[n]\C} iteratively selects the simultaneous distance Z f and Z r The furthest embedding:

[0042]

[0043] Where λ is the balance coefficient, C represents the index of the core set element in Z′, and C f is the index of the potential abnormal core set that is screened out in Z′.

[0044] Compared with the prior art, the present invention has at least the following advantages:

[0045] The industrial abnormal pollution control method based on dual-perspective core filtering proposed in the present invention is suitable for more complex industrial production environments. A dual-perspective model based on feature space is designed to filter potential anomalies and construct a purer core set for application in industrial defect detection. Specifically, based on the prior art on feature extraction and sampling methods, a global and local dual-perspective filtering mechanism is established for the core set obtained by further sampling to filter outliers both globally and locally. Although there are the above dual mechanisms to reduce the loss of normal cores, in order to further ensure the representativeness of the core set, the present invention proposes a scheme of far-point balanced supplementary sampling, which avoids the adjacent infiltration of selected potential anomalies as much as possible while supplementing normal cores. The present invention designs a dual mechanism of potential anomaly filtering and core supplementary sampling. Compared with the existing scheme, more normal and representative core embeddings can be extracted, and the improvement effect of industrial defect detection in non-pure training environments is obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention and the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a simplified flow chart of the method of the present invention.

[0048] Figure 2 is a schematic diagram of the potential anomaly global filtering module.

[0049] Figure 3 is a schematic diagram of the potential abnormal local filtering module.

[0050] Figure 4 It is a schematic diagram of far-point balanced supplementary sampling.

[0051] Figure 5 This is the visualization result of the anomaly filtering effect of some categories in the MVTec-AD dataset.

[0052] Figure 6 It is the calculation method of the anomaly segmentation map. DETAILED DESCRIPTION

[0053] The present invention is described in further detail below.

[0054] The present invention designs an industrial abnormal pollution control method based on dual-view core filtering, which filters the block-level features of abnormal pollution mixed in the unlabeled industrial image set to cope with the real production environment. Specifically, given an unlabeled industrial image training set X = {X1,…,X n} Input to get the final core set The governance framework needs to go through three steps: feature extraction and sampling, core set filtering, and core set supplementary sampling. Represents the index of the core set element, indicating that the core set M is from the embedding set Z = {Z1,…,Z n} is sampled from.

[0055] The present invention will be further described with reference to the accompanying drawings and embodiments. Figure 1 As shown, a method for industrial abnormal pollution control based on dual-view core filtering includes the following steps.

[0056] S1: Feature extraction and core set sampling. Given an unlabeled industrial image training set X = {X1,…,X n}, where X is mixed with real abnormal images X a , where |X a |<<|X|. Most of the deep anomaly detection methods in the past chose to fine-tune pre-trained features to adapt to the task. Existing studies have found that in fact, it is necessary to directly use pre-trained intermediate layer features to apply to industrial detection data, because the semantic information of shallow features is not abstract enough, and deep features are more biased towards pre-trained image sets and have more deviations from the target domain. This paper follows the approach of PatchCore and extracts the corresponding feature tensors from each network layer of WideResNet, denoted by F i is the feature tensor extracted from the i-th network layer of WideResNet, F i The dimension is (b i ,l i ,l i ), b iis the number of feature channels of the i-th network layer, l i F i Regarding the feature map size of the image, F i By b i l i × i For F i A single feature map is taken, and the elements at each position are traversed. The P×P neighbors of each element are sampled. With the help of edge padding operation, a new set of block-level feature maps is obtained from the single feature map, with a total of c i = l i × i indivual;

[0057] The i-th network layer of WideResNet has a dimension of (c i ,b i ,P,P)’s feature tensor F i ', obviously as i increases, the network layer deepens, c i Continuously decreasing, b i Keep increasing. The minimum value of WideResNet network layer i is i min , the maximum value is i max . i 'Along c i and b i Interpolate the two dimensions and convert the dimensions to Then average pooling along the last two dimensions gives a dimension of The aggregate tensor F i ″; Finally, the F corresponding to each layer i ″ are concatenated and pooled, and finally the block-level embedding set Z = {Z1,…,Z m},in, m is the total number of block-level embeddings in Z, and the above feature extraction and aggregation operations are recorded as A single sample X i After P(·) operation, multiple block-level embeddings can be obtained. In the subsequent experimental verification, P takes 3, i min Take 2,i max Take 3.

[0058] For all embeddings Z of all samples extracted above, first use random linear mapping to reduce the dimension to a lower embedding space Z′. Because the similarity of industrial inspection image data is very high, and the high-dimensional semantic vectors obtained in the above process have a lot of redundant information, random linear mapping can be used to successfully compress it. Next, the Core-set algorithm is used to perform core set sampling on the above reduced-dimensional vector set to obtain the index of the core set element, and the core set M to be processed is taken out from the embedding set Z. It should be noted that no elements are selected at the start of the algorithm, so a part of the embedding is randomly selected first, and the average distance of other embeddings to the selected point is calculated. The one with the longest average is added to the initial set to start the Core-set. This method is a greedy means and selects the global optimal point, but it starts quickly and uses less computing resources.

[0059] Based on the embedding set Z, use random linear mapping to reduce the dimension to a lower embedding space to obtain a new embedding set Z′. Then use the Core-set algorithm to sample Z′ and extract the core set to be processed. Represents the index of the core set element in Z′. In the startup phase of the Core-set algorithm, a part of the embeddings in Z′ is randomly selected, the average distance from other embeddings to the selected embeddings is calculated, and the embedding with the farthest average is taken to initialize M. In this way, Core-set is started, and sampling is iterated in a loop until the number of embeddings in M ​​reaches the expected target.

[0060] S2: Based on the assumption that abnormal embeddings are outlier distributions, the present invention selects potential contamination points from the perspectives of embedding overall distribution and local association. Because the abstract features of the pre-trained network are extracted, the input image obtains a series of feature embeddings, which correspond to the abstract semantics of each block-level area of ​​the input. Due to potential abnormal interference, the defective area of ​​the abnormal input is also mapped to this space, and the abnormal embedding is often located in the marginal distribution. The sampling principle of the core set is to select as diverse samples as possible, so it is very likely to hit these block-level features from abnormal images. Starting from the dual perspectives of the whole and local areas, potential block-level abnormal features can be searched more comprehensively and accurately without supervised signals.

[0061] It should be noted that the proposed dual-view filter only needs to be based on the core set itself, without using the entire embedding set Z to calculate the association, which improves the efficiency of the filtering calculation. Secondly, for a single abnormal image, not all of its areas are defective, so its normal block-level features do not actually affect the detection, and can even provide new patterns that were not originally in the training set. Based on this, the present invention only needs to filter features but not samples, and the following filtering operations are only performed at the feature level.

[0062] Based on the core set M obtained from S1, we select the potential abnormal latent set from the perspective of overall distribution and local association.

[0063] S2-1: Global filter. PatchCore is based on the assumption that the feature embeddings of anomalies are far apart in space from normal embeddings, that is, the anomalies themselves are actually sparsely distributed and exist far away from most normal points. Therefore, the anomaly score can be defined based on the distance between the test point and the typical point in the core set. Following the sampling process of PatchCore, the algorithm must select the point farthest from the core set from the remaining candidate points each time. Therefore, once an anomaly point is selected, other anomaly points in its vicinity will be difficult to hit in subsequent selections. Different types of anomalies are also quite different, and they do not affect each other during the selection process, and can be seen as presenting a multi-cluster distribution. Overall, the distribution of the core set converges with the distribution of all embeddings. Therefore, the present invention believes that the global filtering algorithm only needs to observe the distance distribution of the core set alone.

[0064] S2-1-1: Use global filters to establish global associations between the elements of M, and set thresholds to find potential anomalies that are far away from the vast majority of points and index them Record, where n represents the total number of samples in the training set X.

[0065] In fact, there are fewer points in the core set that are closest to the abnormal core, and the abnormal points are still scattered as a whole. In order to quantify this relationship, we first use the distance between any two core embeddings to define the similarity between them. Specifically, for each core embedding M i , first find out how to embed M with other cores j The Euclidean distance similarity is:

[0066]

[0067] Here, softmax is used for normalization in order to construct an asymmetric similarity matrix. Because for any two points Z′ i and Z′ j (i≠j), they are not necessarily neighbors of each other, so Z i With Z j The similarity between them is not equal to Z j With Z i The similarity of needs to be compared with the distance of other points to get their respective one-way similarities. Here, D(·,·) is used to represent the similarity between the two points, and the following formula is used for unified calculation:

[0068]

[0069] Among them, ‖·‖2 represents the l2 norm, corresponding to the calculation of the Euclidean distance. ∈ represents the minimum value, which prevents the calculation error of the denominator being zero, corresponding to the distance between the two points being too close. And Z′ i The similarity with itself is set to negative infinity to prevent it from interfering with the normalization calculation.

[0070] S2-1-2: Based on the above analysis, the algorithm can obtain the similarity matrix Sim of the unidirectional relationship, where each row i represents Z′ i Next, sum each column j and calculate the similarity of other points to Z′ j The total similarity of can be regarded as the overall association of other members with the core. All the accumulated results are sorted from small to large, and the embeddings in the top α% are selected and defined as global potential anomalies:

[0071]

[0072] ArgLastK(·,α) means taking the index of the last α% of members. Figure 2 As shown in the figure, the gray solid points in the two-dimensional space represent normal core embeddings, while the gray circled points are embeddings from abnormal samples. Even in the sampled core set, the number of abnormal embeddings is still sparse compared to normal points. In the final selected global relationship candidate set, most of the easily identifiable anomalies will be included.

[0073] S2-2: Local filter. After the global relationship screening between the core set embeddings, the pre-eliminated embeddings actually include two situations: outliers and edge points of normal distribution, where the latter refers to normal embeddings with high recognition difficulty, corresponding to features with low frequency in normal images. If the number of such difficult normal embeddings filtered is not reduced, it is easy to cause similar rare normal features that appear in the test to be misjudged, because the original recent embedding does not exist in the core set, and is replaced by other core embeddings, which distances similar normal features from the core set, and the abnormal score becomes higher. The commonality between outliers and difficult normal points is that they deviate from the central main cluster as a whole. The difference is that the former mostly presents an island distribution, and the latter is mainly located in the peninsula or marginal area where the main cluster extends. Therefore, based on such prior knowledge, clustering can be used to study local cluster distribution. However, traditional clustering does not deliberately distinguish between isolated clusters and peninsula clusters. The present invention specifically divides peninsula clusters together with the main cluster, so a small cluster screening method based on local association is designed. On the basis of initializing a series of small clusters, by comparing the density of each cluster and continuously merging adjacent clusters, we finally get several main clusters and discrete small clusters, and filter the small clusters with a smaller number as a whole.

[0074] Use local filters to first divide the edge points and the main cluster into the same subset. On the basis of initializing a series of small clusters, by comparing the density of each cluster, continuously merging adjacent clusters, finally obtaining several main clusters and discrete small clusters, and filtering the small clusters as a whole. The index of the selected core is recorded as

[0075] S2-2-1: We only need to focus on the correlation between the internal elements of the core set, cluster them first, and divide the embeddings into multiple small clusters with fewer internal points. Without the help of other algorithms, the present invention follows the sampling method of PatchCore, selects the core again for the core set M obtained for the first time, and uses the preferentially selected embeddings as cluster centers to obtain a smaller core set, while other core set members are assigned to corresponding clusters according to their nearest neighbors. In order to avoid some isolated discrete points being mistakenly assigned to the nearest main cluster due to too few cluster centers, it is necessary to set as many cluster centers as possible to subdivide different small clusters. To achieve this goal, sample γ% of the cluster centers of the core set M, and obtain about 1 / γ% core embeddings per cluster on average. In the worst case, there may be only one cluster center.

[0076] S2-2-2: Next, the local filtering algorithm needs to merge the small clusters obtained in the previous step, merging the adjacent small clusters into one, so that the peninsula cluster can be merged into the main cluster. Specifically, the "cluster density" ρ is calculated for each small cluster, which is defined as the average distance between all embeddings in the current cluster and the nearest neighbors in the cluster, that is,

[0077]

[0078] Here Q represents the index set of the core embedding in the current cluster. When there is only one cluster member, ρ = 0. Then find the k nearest cluster centers according to the distance between each cluster center. Each cluster only compares the correlation with its neighbor cluster. The principle of comparison is to find the "cluster density" ρ within the two clusters. i and ρ j , if the closest distance between two clusters (i.e. the closest distance between all points in the two clusters) d is greater than ρ i and ρ j If both are small, it means that they belong to the same distribution, then the two clusters are merged and the new index set is recorded as Q′ i =Q i ∪Q j , the total number of new clusters is recorded as m′. Figure 3 As shown in the figure, when calculating the "cluster density" within the original cluster, each point only focuses on the embedding point closest to itself, and does not consider the farthest distance because it prevents some clusters from being too numerous when they are first divided, and the farthest distance will interfere with the calculation of density. The hollow circle represents the closest point between clusters, which builds a bridge to connect two different clusters.

[0079] S2-2-3: The above merging process only needs one round, because each cluster only needs to focus on the association with the neighboring clusters. When the cycle is over, the entire core set will build a different network structure. At this time, the number of embeddings of each cluster is compared with the total number of core sets. Clusters that are no more than β% are defined as outliers, and their embedding index is

[0080]

[0081] C l The corresponding embeddings in the initial core set M are local potential anomalies.

[0082] S2-3: Dual filter cooperation. The two filters work independently, without any order or mutual coupling in input and output. They are both based on the complete unfiltered core set for screening, because they consider each other from different angles, but both need to use the entire initial core set M. Finally, the intersection of the two screened candidate indexes is obtained, that is,

[0083] C f =C g ∩C l .

[0084] According to C f The index in M ​​is used to extract the corresponding embedding from M, which constitutes the potential abnormal latent set M a It should be noted that we do not use global filters or local filters separately, but make judgments under global and local relationships at the same time. This method can reduce and avoid more normal misscreening.

[0085] S3: Due to the lack of supervision information, the filtered core set is likely to lack normal core points. Especially in the case of less abnormal noise, directly applying a single-class model is sufficient for detection. Therefore, the present invention additionally proposes a method of supplementing the core set driven by potential anomalies. Figure 4 As shown in the figure, different clusters can be formed in the high-dimensional hyperspace with each core set element as the center. The hollow circles represent normal clusters, and the gray dots represent clusters centered on abnormal embeddings. Similar anomalies are more likely to be concentrated in this area. In order to avoid the new embeddings that continue to be sampled being too close to the filtered anomalies, the algorithm needs to select as few points as possible within its cluster, otherwise the meaning of embedding filtering will be lost. At the same time, it is hoped that the supplementary points will be different from the normal embeddings that have been filtered out, that is, to expand its diversity as much as possible based on the remaining core sets. However, these two goals actually need to be balanced with each other, and it is difficult to obtain the embedding that is farthest from both at the same time every time.

[0086] Supplement sampling for some lost normal cores, the number of samples is the same as M aThe sampling method is iteratively expanded, and each time an embedding is selected from the original embedding set Z′, which is the same as M a and the remaining cores are embedded in M\M a The furthest embedding is added to the core set, and the above steps are repeated until the predetermined number is reached to obtain the final core set M′.

[0087] The process of supplementary sampling is as follows: based on the core set Z after screening r = {Z′ j |j∈C\C f} and the potential abnormal core set Z that is screened out f = {Z′ j |j∈C f}, from the remaining embedding Z fr = {Z′ i |i∈[n]\C} iteratively selects the simultaneous distance Z f and Z r The furthest embedding:

[0088]

[0089] Where λ is the balance coefficient, C represents the index of the core set element in Z′, and C f is the index of the potential anomaly core set that is screened out in Z′. In the actual verification process, λ is set to 0.1, because the number of screened out anomalies is small, which prevents them from causing excessive interference to the supplement of diversity.

[0090] S4: In the inference phase, for a single new test image x test Use the method described in S1 The operation extracts features and aggregates them to obtain the embedding of each block level Then calculate each block level embedding z based on M′ test The anomaly score, z test The closest distance to the embedding in M′, all block-level embeddings z test The maximum value of the anomaly score is defined as x test The anomaly score s is:

[0091]

[0092] Compared with the image-level scores, each block-level score constitutes a low-size anomaly segmentation map h. In order to facilitate the comparison with the original pixel set annotation, the block-level segmentation map needs to be divided into Figure 6 The above steps are operated, and a segmentation map H of the same size as the original input is obtained through bilinear interpolation, and then smoothed by a Gaussian filter (σ=4), and finally a smoothed version of the segmentation map H′ is obtained. Among them, the brighter the pixel, the more likely it is to be in a defective position, and vice versa, the more likely it is to be normal.

[0093] The present invention uses the MVTec-AD dataset to verify the rationality of the invention content. The MVTec-AD dataset is widely used in various industrial image defect detection. It contains a total of 15 categories of sub-datasets, including 5 texture categories (carpet, grid, leather, tile and wood) and 10 object categories (glass bottle, cable, capsule, hazelnut, metal nut, pill, screw, toothbrush, transistor and zipper), and the samples of each training set range from 60 to 391. Following most industrial defect detection studies, AUC is used to measure the abnormal score predicted by the model, and the AUC value needs to be compared at both the instance level and the pixel level. For ease of distinction, the instance level AUC is recorded as IAUC, the pixel level AUC is recorded as PAUC, and the percentage (%) is taken to record. This section selects PatchCore (PC), SPADE and SimpleNet (SN) as the comparison objects of the present invention. The three methods are defect detection methods based on pre-training embedding. PatchCore is based on sampling of the core set and is obtained by comparing the distance in the embedding space, based on the features obtained by deep pre-training. SPADE extends the nearest neighbor retrieval from instance-level recognition to sub-graph-level recognition, and obtains classification results for different regions. SimpleNet uses PatchCore’s feature extractor to add Gaussian noise to normal embeddings to generate pseudo-abnormal embeddings, and then trains a discriminator to identify anomalies.

[0094] Table 1. Comparison of instance-level detection performance (IAUC) on the MVTec-AD dataset

[0095]

[0096]

[0097] Since MVTec-AD only has normal training samples, in order to create a training set with abnormal noise, the verification experiment randomly samples a certain number of counterexample images from the test set and mixes them with the original training set. Among them, the ratio of normal to abnormal samples in the new training set is set to 1:η (η<1). Since the ratio of normal to abnormal samples in the test set is destroyed, this chapter sets two different settings to test the performance of the model:

[0098] Original formula (ORG): Use the original test set because the ratio of normal and abnormal samples is appropriate, and the image-level AUC result is more meaningful. At the same time, the present invention and its comparison method both follow a completely unsupervised training environment. Although the training set and the test set have overlapping parts, the model does not know the true label of the sampled abnormal samples, which is equivalent to direct learning, so as to verify the impact of abnormal noise on model performance.

[0099] Inductive (IND): Another approach is to follow the commonly used inductive comparison. The data sampled for training is no longer inferred. The test set consists of the remaining samples after sampling and does not overlap with the training set.

[0100] Table 2. Comparison of pixel-level detection performance (PAUC) on the MVTec-AD dataset

[0101]

[0102]

[0103] Table 1 shows the instance-level detection results (IAUC) of the present invention and other comparative methods on all categories of the MVTec-AD dataset, where the proportion of abnormal noise in the total training set η is 0.1, the best result on each category is marked in bold, and the second is underlined (the same applies to the following tables). It is observed that the method proposed in the present invention achieves the best detection performance in both settings. On the original test set, the new method has a significant improvement (6.62% higher than the baseline method PatchCore), and the advantage of eliminating a small number of potential anomalies from the core set is brought into play. In particular, although SimpleNet is better than PatchCore at this time, the present invention has obvious improvements on it and performs better. Similarly, under the inductive setting, the method of the present invention is still strong, with better detection performance on up to 11 categories. Regardless of the test setting sampled, the proposed method ranks first in almost all texture categories. The results show that the performance of instance-level defect detection is significantly improved after the present invention eliminates potential abnormal features.

[0104] Table 2 shows the performance of each method on the pixel-level defect detection task (PAUC), which counts the anomaly score inference performance of all pixels of all test samples. First, in the original reasoning setting, although the proposed method is not as good as SimpleNet, the difference between the two is not much. Because the core set is sampled from the features of the training set, which is used for reasoning, the polluted embedding space is more destructive to it. However, compared with the baseline version PatchCore, the improvement of the present invention is actually effective and feasible. In summary, the methods based on core set sampling (the present invention and PatchCore) are obviously effective, higher than SimpleNet and SPADE, reflecting the high performance detection of this type of method. At the same time, the present invention is further improved on the basis of PatchCore.

[0105] Figure 5To further study the effect of the proposed core embedding filtering, glass bottles, hazelnuts, carpets, and tiles on the MVTec-AD dataset are selected for visualization, which are representatives of object classes and texture classes, respectively. Block-level feature representatives are sampled from the two types of training sets, and the high-dimensional core embedding is mapped to a two-dimensional space using the t-SNE tool, and the result is plotted as shown in the figure. Among them, the number 0 represents the normal core embedding (NE). Since not all areas of the abnormal image are defects, the core embedding taken from it is called anomaly candidate (AC), which is plotted using the number 1. The light gray and dark gray hollow circles correspond to the filtered normal embedding (FN) and anomaly candidate (FA), respectively. First, by only observing the relationship between the selected samples and the overall data distribution, it can be clearly found that in the two types of data, the global associator does not only focus on the discretely distributed embedding points, but also selects a part of the main cluster points. The local associator is more focused on areas with low relationship density, especially small clusters that are not associated with the surrounding areas. Secondly, on the glass bottle class, the global filter performs better than the local filter. Both of them mistakenly filter out a similar number of normal embeddings, but the global filter can find more anomaly candidates, while the latter finds much fewer. This is because the image feature patterns of this object class are more, and the corresponding embedding space presents different main clusters. There is a lack of clusters with overwhelming numbers, and the local associator is prone to failure. At the same time, the object class occasionally has logical anomalies, such as changes in the placement of objects. Such abnormal features are connected to the main cluster and do not exist in isolation. Using global association to study the similarity between most normal pattern points and it helps to discover the location of such anomalies. Hazelnuts, which are also an object class, are easy to distinguish because of their obvious defect characteristics, such as cracks and holes. Therefore, local anomalies can separate their corresponding isolated embeddings. Observing the carpet and tile images, it can be found that their feature patterns are relatively simple, repeated textures are spread throughout the image, and the corresponding embedding space distribution is also relatively concentrated. The defects in this type of data are also mostly structural anomalies, such as cracks, oil stains, etc., and the corresponding features are isolated from the main cluster of textures in the embedding space. At this time, the local associator is more effective. Compared with separate global and local associators, using both can avoid more normal elements from being mistakenly screened out, while retaining most of the main anomaly candidates. Although the present invention uses secondary sampling to supplement some cores, it is still difficult to select points near difficult normal embeddings based on the balance strategy, and can only alleviate the loss of normal cores to a certain extent. When the global and local associations are integrated, the hit rate of abnormal candidates will be greatly improved.

Claims

1. The industrial abnormal pollution control method based on dual-perspective core filtering is characterized by: The steps include: S1: Feature extraction and core set sampling: Given an unlabeled industrial image training set X = {X1,…,X n }, where X is mixed with real abnormal images X a , where |X a |<<|X|; Following the approach of PatchCore, the corresponding feature tensors are extracted from each network layer of WideResNet, and recorded as F i is the feature tensor extracted from the i-th network layer of WideResNet, F i The dimension is (b i ,l i ,l i ), b i is the number of feature channels of the i-th network layer, l i F i Regarding the feature map size of the image, F i By b i l i ×l i The feature map composition of size; For F i A single feature map is taken, and the elements at each position are traversed. The P×P neighbors of each element are sampled. With the help of edge padding operation, a new set of block-level feature maps is obtained from the single feature map, totaling c i =l i ×l i indivual; The i-th network layer of WideResNet has a dimension of (c i ,b i ,P,P)’s feature tensor F i ', obviously as i increases, the network layer deepens, c i Continuously decreasing, b i Keep increasing; remember that the minimum value of WideResNet layer i is i min , the maximum value is i max ; for F i 'Along c i and b i Interpolate the two dimensions and convert the dimensions to Then average pooling along the last two dimensions yields a dimension of The aggregate tensor F i ″; Finally, the F corresponding to each layer i ″ is concatenated and pooled, and finally the block-level embedding set Z={Z1,…,Z m },in, m is the total number of block-level embeddings in Z, and the above feature extraction and aggregation operations are recorded as A single sample X i go through The operation can get multiple block-level embeddings. Based on the embedding set Z, use random linear mapping to reduce the dimension to a lower embedding space to obtain a new embedding set Z′. Then use the Core-set algorithm to sample Z′ and extract the core set to be processed. Represents the index of the core set element in Z′. In the startup phase of the Core-set algorithm, a random portion of embeddings in Z′ is selected, and the average distance from the other embeddings to the selected embedding is calculated. The embedding with the farthest average distance is used to initialize M. This is used to start the Core-set. The sampling is iterated in a loop until the number of embeddings in M ​​reaches the expected target. S2: Based on the core set M obtained in S1, we select the potential abnormal latent set from the perspective of overall distribution and local correlation. S3: Supplement sampling for some lost normal cores. The number of samples is the same as M a The sampling method is iteratively expanded, and each time an embedding is selected from the original embedding set Z′, which is the same as M a and the remaining cores are embedded in M\M a For the farthest embedding, add the embedding to the core set and repeat the above steps until the predetermined number is reached to obtain the final core set M′; S4: For a single new test image x test Use the method described in S1 The operation extracts features and aggregates them to obtain the embedding of each block level Then calculate each block level embedding z based on M′ test The anomaly score, z test The closest distance to the embedding in M′, all block-level embeddings z test The maximum value of the anomaly score is defined as x test The anomaly score s:

2. The method for industrial abnormal pollution control based on dual-view core filtering according to claim 1, characterized in that: The potential abnormal latent set M is selected from S2 a The process is as follows: S2-1: Use global filters to establish global associations between elements of M, and set thresholds to find potential anomalies that are far away from the vast majority of points and index them Record, where n represents the total number of samples in the training set X; S2-2: Use local filters to first divide the edge points and the main cluster into the same subset. On the basis of initializing a series of small clusters, by comparing the density of each cluster, continuously merge adjacent clusters, and finally obtain several main clusters and discrete small clusters. Perform overall filtering on the small clusters with a smaller number. The index of the selected core is recorded as S2-3: Find the intersection of the candidate indexes selected by the global filter and the local filter, that is, C f =C g ∩C l According to C f The index in M ​​is used to extract the corresponding embedding from M, which constitutes the potential abnormal latent set M a .

3. The method for industrial abnormal pollution control based on dual-view core filtering according to claim 2, characterized in that: The S2-1 global filter includes the following steps: For each core embedding M i , first find out how to embed M with other cores j The Euclidean distance similarity is: Among them, softmax represents normalization processing, and D(·,·) is used to represent the similarity between the two, that is: Where ‖·‖2 represents the l2 norm, corresponding to the calculation of the Euclidean distance, ∈ represents the minimum value, and Z′ i The similarity with itself is set to negative infinity; based on this, the similarity matrix Sim of the one-way relationship is obtained, and each row i represents Z′ i Relative to the similarity of other points, sum each column j and statistically calculate the similarity of other points to Z′ j The total similarity of , all cumulative results are sorted from small to large, and the top α% of the partial embeddings are selected and defined as global potential anomalies: ArgLastK(·,α) represents the index of the member that ranks in the last α%.

4. The method for industrial abnormal pollution control based on dual-view core filtering according to claim 3 is characterized in that: The S2-2 local filter comprises the following steps: The core set M is further sampled using the PatchCore algorithm to obtain a smaller number of core sets. γ% of the cores are selected and embedded as the cluster centers of M. The unselected members of M are assigned to the clusters corresponding to their nearest neighbors. On average, each cluster obtains about 1 / γ% core embeddings. The closest distance d between two clusters is less than the "cluster density" ρ corresponding to the two clusters. i and ρ j , then merge the two clusters into one so that the edge is embedded into the main cluster, and the new index set is recorded as Q′ i =Q i ∪Q j , the total number of new clusters is recorded as m′, and the cluster "cluster density" ρ is calculated as follows: Where Q represents the index set of core embeddings in the current cluster when there is only one cluster member; The above merging process is only performed once. The number of embeddings of each cluster is compared with the total number of core sets. Clusters that are no more than β% of the total number are defined as local potential anomalies: Among them C l The corresponding embeddings in the initial core set M are local potential anomalies.

5. The method for industrial abnormal pollution control based on dual-view core filtering according to claim 4 is characterized in that: The supplementary sampling in S3 includes the following steps: Based on the core set Z after screening r ={Z′ j |j∈C\C f } and the potential abnormal core set Z that is screened out f ={Z′ j |j∈C f }, from the remaining embedding Z fr ={Z′ i |i∈[n]\C} iteratively selects the simultaneous distance Z f and Z r The furthest embedding: Where λ is the balance coefficient, C represents the index of the core set element in Z′, and C f is the index of the potential abnormal core set that is filtered out in Z′.

Citation Information

Patent Citations

  • Active anomaly detection method for driving multi-expert mixed training by data independent subsets

    CN117648656A

  • Image defect detection method based on double-branch inverse distillation and multi-input image

    CN118521570A

  • Systems and methods for semiconductor defect-guided burn-in and system level tests

    US20220390505A1