An active learning anomaly detection method and system based on graph propagation

Abstract

The present invention discloses an active learning anomaly detection method and system based on graph propagation, which relates to the technical field of image anomaly detection. After collecting product images, they are preprocessed, and a pretrained autoencoder is constructed and trained. The trained autoencoder obtains image encoding that does not contain low-level features and is further reconstructed, thereby achieving not only a more continuous and smooth low-dimensional representation of the image, but also better anomaly detection performance. Graph propagation based on the k-nearest neighbor propagation matrix updates image annotation information, making it less time-consuming to update image annotation information. In the active learning stage, the uncertainty and representativeness of image samples are simultaneously considered, and the most representative image samples are selected. The image sample subspace is fully explored, effectively improving the performance of active learning. Finally, the product image to be detected is input into the anomaly detection model to obtain the anomaly score of the product to be detected, with excellent anomaly detection performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image anomaly detection, and more specifically, to a graph propagation-based active learning anomaly detection method and system. Background Art

[0002] With the increasing automation in the manufacturing industry, pre-emptive product testing is becoming increasingly important in real-world manufacturing scenarios, as is the need to identify and recall substandard products to meet industry standards and ensure the overall quality of shipped products. Automated optical quality inspection systems are becoming increasingly important in these scenarios. In these real-world manufacturing scenarios, it is often necessary to capture and annotate partial product images to determine the distribution of sample patterns. Based on this distribution, a suitable anomaly detection model is then trained to detect these patterns. The performance of these anomaly detection models has a crucial impact on the performance of automated optical quality inspection systems.

[0003] Compared with ordinary detection tasks, the number of abnormal images that can be collected in manufacturing environments is relatively small, and in practical applications, it is usually impossible to summarize abnormal patterns, which limits the use of current supervised learning-based methods in anomaly detection.

[0004] Existing research typically defines anomaly detection as a semi-supervised anomaly detection problem. This approach attempts to obtain an approximate distribution of normal samples at the expense of a small amount of labeling effort. Machine learning algorithms are then used to measure the scores of samples to be tested that conform to the approximate distribution, effectively detecting anomalies. Based on the labeling method, existing research can be divided into two main categories: 1) Randomly selecting and labeling a fixed budget of samples from the collected data to obtain an approximate distribution of normal samples. However, many of the selected samples are not located at the edge of the distribution, resulting in a fuzzy distribution edge that affects detection accuracy. 2) Active anomaly detection methods utilize active learning to select samples from the sampled data that are likely to be at the edge of the distribution. This approach alleviates the problems of the first category. However, current active anomaly detection methods still face two challenges in image tasks: low detection performance and time-consuming updating of the sample selection interval, which seriously hinders their practicality. Summary of the Invention

[0005] To solve the problems of low detection performance and long image sample updating in current active anomaly detection methods, the present invention proposes an active learning anomaly detection method based on graph propagation, which has high anomaly detection performance, uses the nearest neighbor propagation graph to update image annotation information, and has a short updating time.

[0006] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0007] A graph propagation-based active learning anomaly detection method, the method comprising the following steps:

[0008] S1. Randomly collect images of several products and preprocess them to obtain a product image set;

[0009] S2. Construct a pre-trained autoencoder. Use the reconstruction error and L2 regularization as the objective function and train the pre-trained autoencoder using the product image set until convergence. Training is complete, and the autoencoder is obtained.

[0010] S3. Use an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and construct a k-nearest neighbor propagation matrix as the adjacency matrix of the graph structure for edge weight propagation;

[0011] S4. Select the most representative image samples for manual annotation, and use the manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure;

[0012] S5. Utilize the edge weights of the graph structure to propagate uncertainty and representativeness of each node to neighboring nodes, and select image samples for active learning using a heuristic sample selection strategy. This process is iterated until the annotation budget is exhausted and active learning ends.

[0013] S6. Calculate training weights for each image sample using the uncertainty and adjacency of the nodes, and retrain the pre-trained autoencoder with the weights to serve as an anomaly detection model.

[0014] S7. Collect image data of the product to be detected and perform the preprocessing described in S1. Finally, input the data into the anomaly detection model and use the reconstruction error to calculate the anomaly score of the product to be detected.

[0015] This technical solution first collects product images, then preprocesses them, constructs a pre-trained autoencoder, and trains it. The autoencoder obtained through training first obtains image encoding that does not contain low-level features, and further reconstructs it, which not only achieves a more continuous and smooth low-dimensional image representation, but also achieves better anomaly detection performance. Then, the image annotation information is updated based on the graph propagation of the k-nearest neighbor propagation matrix. Since the relationship between images has been modeled when the k-nearest neighbor propagation matrix was constructed, the time consumption of updating the image annotation information is reduced. In addition, the uncertainty and representativeness of the image samples are considered simultaneously in the active learning stage, and the most representative image samples are selected. The image sample space is fully explored, which effectively improves the performance of active learning. Finally, the product image to be detected is input into the anomaly detection model to obtain the anomaly score of the product to be detected, with excellent anomaly detection performance.

[0016] Preferably, in step S1, n unlabeled images are randomly collected from real manufacturing scenes and preprocessed, wherein the preprocessing includes template matching, cropping, and normalization to obtain a product image set, which is denoted as D u ={x1,x2,…,x n}, where x i ∈R c×h×w , c, h, and w are the number of channels, height, and width of the image respectively.

[0017] Preferably, the pre-trained autoencoder comprises an encoder and a decoder, wherein the encoder comprises a pre-trained convolutional neural network and a multi-layer perceptron connected to the pre-trained convolutional neural network, wherein the pre-trained convolutional neural network encodes the input image locally but not including low-level features, and then the multi-layer perceptron performs a low-dimensional representation on the encoded image;

[0018] The decoder includes a multi-layer perceptron and a deconvolutional neural network connected to the multi-layer perceptron. The low-dimensional representation is input into the multi-layer perceptron of the decoder, and the multi-layer perceptron performs a one-time upsampling on it. The upsampling result is input into the deconvolutional neural network, and the deconvolutional neural network reconstructs the original image.

[0019] Here, through the use of a pre-trained autoencoder structure, the low-level features of the input image are gradually discarded during dimensionality reduction. During training, the high-level features of the image are always mapped to the low-level features of a normal image. However, in industrial anomaly detection tasks, product anomalies are often localized and are more often expressed as low-level features of the image. Therefore, the reconstructed image of an abnormal image after the pre-trained autoencoder has normal low-level features, which results in higher reconstruction error for abnormal image samples, making it better suited for anomaly detection. Furthermore, the pre-trained autoencoder has strong continuity and smoothness, which are largely retained after further dimensionality reduction. Therefore, better image encoding is achieved, resulting in a better low-dimensional image representation for end-to-end anomaly detection.

[0020] Preferably, in step S2, the objective function is expressed as:

[0021]

[0022] in,

[0023]

[0024]

[0025] Among them, J c (θ,γ) represents the reconstruction error; J r(θ,γ) represents the L2 regularization term; D represents the decoder, θ represents the hyperparameter corresponding to the decoder; E represents the encoder, γ represents the hyperparameter corresponding to the encoder; λ is a predefined hyperparameter; x i represents the i-th image in the product image set; |.|_2 represents the L2 norm;

[0026] During the training process, the parameters of the pre-trained convolutional neural network are fixed to ensure that the input obtained from the training part corresponding to the unfixed parameters does not contain low-level features of the image but retains local features. The product image set is divided into batches of fixed size and input into the pre-trained autoencoder. The loss error of the objective function is back-propagated through small batches until the loss error no longer decreases significantly. The training is completed and the autoencoder is recorded as D(E(x i ,γ),θ).

[0027] During pre-trained autoencoder training, the product image set is divided into batches of fixed size and fed into the pre-trained autoencoder. The loss is calculated (because the minimum value of the objective function is 0, the loss error is the value of the objective function at 0). The loss is then backpropagated through small batches until the loss no longer decreases significantly, at which point the training converges. The trained autoencoder can reduce the image dimension, encoding the locations where the encodings are placed as node positions in a graph structure, with edges forming the graph structure between the nodes. The autoencoder can then reconstruct the reduced-dimensional image.

[0028] Preferably, in step S3, the encoding and reconstruction errors of each image in the product image set obtained by the autoencoder are respectively recorded as V = {v1, v2, ..., v n} and ε={e1,e2,…,e n}, v i and e i The calculation expressions are:

[0029] v i =E(x i ,θ)

[0030] t i =|x i -D(v i ,γ)|2

[0031]

[0032] Among them, v i 、e i are the encoding error of the i-th image in the encoder and the reconstruction error of the i-th image in the pre-trained automatic encoder; t i Represents the unnormalized reconstruction error, described as an intermediate variable;

[0033] Let the constructed k-nearest neighbor propagation matrix be expressed as S, which is expressed as:

[0034] S=D -1 / 2 W s D -1 / 2

[0035]

[0036] Where D is the degree matrix and the diagonal is the symmetric affine matrix W s The sum of the corresponding rows, other positions are 0; symmetric affine matrix W s The calculation expression is:

[0037]

[0038] Among them, W size is n×n, and the k-nearest neighbor propagation matrix is ​​calculated using the radial kernel function. The k-nearest neighbor distance W between the i-th and j-th samples is ij The calculation expression is:

[0039]

[0040] Among them, knn(i) represents the set of k nearest neighbor samples of sample i.

[0041] Preferably, in step S4, the most representative image samples are selected for manual annotation using the spectral clustering method, and the k-nearest neighbor propagation matrix s is used as the input of the spectral clustering method to cluster all image samples into K clusters, denoted as C = {C1, C2, ..., C K}, and select samples with the minimum encoding distance from other samples in each cluster for manual annotation, and the partially annotated image dataset is recorded as D l ={(x1,y1),(x2,y2),…,(x K ,y K )}, where y i ∈{0,1}, where 0 and 1 are normal and abnormal labels respectively; initialize D u The information node f corresponding to each image sample in ui ,f ri ], as nodes of the propagation graph, where:

[0042] f ui =e i

[0043]

[0044] f ui and f ri Represent the uncertainty and representativeness of the i-th sample respectively, and use the information nodes to construct the information matrix recorded as

[0045] Preferably, the specific process of step S5 is:

[0046] S51. Let the current information matrix be F 0 , the representativeness and uncertainty of the nodes are propagated iteratively through the graph propagation, and the t-th step of the iteration is:

[0047] F t+1 =αSF t +(1-α)F 0 ;

[0048] Until |F t+1 -F t |2<β, where β is a predefined hyperparameter; α=[α1,α2,…,α N ] is used to control the degree of information retention of each node, and the calculation expression is:

[0049]

[0050] Among them, α l and α u are all predefined hyperparameters; let the last step of the iteration be t e , and execute To complete the spread of uncertainty and representativeness;

[0051] S52. Execute heuristic sample uncertainty selection strategy: According to f ui Arrange each unmarked node from high to low, where the first N p The samples corresponding to each node are selected for manual labeling;

[0052] If no abnormal sample is found, the heuristic representative sample selection strategy is executed, and the sample with the largest -f is selected from the unlabeled nodes. ri The samples are manually marked. If abnormal samples are found, or N consecutive p If the heuristic sample selection strategy does not find any abnormal samples, the heuristic sample uncertainty selection strategy is re-executed, and S52 is repeatedly executed until the labeling budget is exhausted.

[0053] This scheme, a strategy for updating annotation information and selecting samples during active learning, leverages graph structures to propagate the uncertainty and representativeness of samples. Compared to other active learning strategies, this one-time modeling of sample relationships using graph propagation occurs before expert annotation. Therefore, even with increasing sample dimensionality, the time required to update annotation information does not significantly increase. Furthermore, this proposal utilizes graph structures to simultaneously propagate both representativeness and uncertainty, which aligns with prior consistency assumptions and has very low computational complexity. As a result, it effectively avoids the phenomenon of missed clusters common in active anomaly detection while maintaining very low computational time, enabling a more comprehensive exploration of the entire sample space and achieving better active learning performance.

[0054] Preferably, after each manual annotation, the node information of the propagation graph is updated using the following formula:

[0055]

[0056] Uncertain sample selection strategy each selects N p After each sample is labeled or each time a sample is selected for labeling by the representative sample selection strategy, S51 is used to propagate the labeling information and update the uncertainty and representativeness of the graph nodes.

[0057] Preferably, let Ω=[ω1,ω2,…,ω N ],ω i ∈R is the training weight, where ω i The calculation expression is:

[0058]

[0059] Among them, F u =[f u1 ,f u2 ,…,f un ];

[0060] The pre-trained autoencoder is retrained using the training weights, and the objective function of the training is recorded as:

[0061]

[0062] in:

[0063]

[0064]

[0065] The training is performed until the objective function converges, and the trained autoencoder is used as the anomaly detection model.

[0066] Preferably, the product image set that collects image data of the product to be inspected and performs the preprocessing described in S1 is denoted as xtest , abnormal score Score(x test ) is calculated as:

[0067] Score(x test )=|x test -D(E(x test ,γ l ),θ l )|2

[0068] Set an empirical threshold Abnormal score Score(x test ) is greater than t, the image sample is identified as abnormal and recalled, and the abnormal score Score(x test ) is less than t and is considered normal.

[0069] This application also proposes an active learning anomaly detection system based on graph propagation, the system comprising:

[0070] The image acquisition and preprocessing module randomly collects images of several products and performs preprocessing to obtain a product image set;

[0071] The autoencoder construction training module is used to construct a pre-trained autoencoder. The reconstruction error and L2 regularization are used as the objective function. The pre-trained autoencoder is trained using the product image set until convergence. The training is completed and the autoencoder is obtained.

[0072] The graph propagation construction module uses an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and constructs a k-nearest neighbor propagation matrix as the edge weight propagation of the graph structure;

[0073] The information matrix construction module is used to select the most representative image samples for manual annotation, and use manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure;

[0074] The active learning module uses the edge weights of the graph structure to propagate the uncertainty and representativeness of each node, and selects image samples for active learning through a heuristic sample selection strategy until the annotation budget is exhausted and active learning ends;

[0075] The anomaly detection module uses the uncertainty of the nodes and the edge weight to calculate the training weight for each image sample, and retrains the pre-trained autoencoder with the weights to serve as the anomaly detection model;

[0076] The image data of the product to be detected is collected and preprocessed, and finally input into the anomaly detection model, and the anomaly score of the product to be detected is calculated using the reconstruction error.

[0077] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0078] The present invention proposes an active learning anomaly detection method and system based on graph propagation. First, product images are collected, then preprocessed, a pretrained autoencoder is constructed and trained, and the trained autoencoder is used to first obtain image encoding that does not contain low-level features. This is then further reconstructed, resulting in not only a more continuous and smooth low-dimensional representation of the image, but also better anomaly detection performance. Next, image annotation information is updated based on graph propagation using a k-nearest neighbor propagation matrix. Because the interrelationships between images are modeled when the k-nearest neighbor propagation matrix is ​​constructed, updating image annotation information is much less time-consuming. Furthermore, the active learning stage simultaneously considers the uncertainty and representativeness of image samples, selects the most representative image samples, and fully explores the image sample subspace, effectively improving the performance of active learning. Finally, the product image to be detected is input into the anomaly detection model to obtain the anomaly score of the product to be detected, achieving excellent anomaly detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A schematic diagram showing a flow chart of the active learning anomaly detection method based on graph propagation proposed in Example 1 of the present invention;

[0080] Figure 2 A schematic diagram showing the structure of the pre-trained autoencoder proposed in Example 2 of the present invention;

[0081] Figure 3 It represents the active learning anomaly detection system based on graph propagation proposed in Example 4 of the present invention. DETAILED DESCRIPTION

[0082] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0083] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0084] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0085] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0086] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0087] Example 1

[0088] like Figure 1As shown in the flowchart, this embodiment proposes an active learning anomaly detection method based on graph propagation, which includes the following steps:

[0089] S1. Randomly collect images of several products and preprocess them to obtain a product image set;

[0090] S2. Construct a pre-trained autoencoder. Using reconstruction error and L2 regularization as objective functions, train the pre-trained autoencoder using the product image set until convergence. Training is complete, resulting in an autoencoder. The trained autoencoder can reduce the dimensionality of the image, encoding the locations where the encodings land as node positions in a graph structure, with edges forming between the nodes. The autoencoder then reconstructs the reduced-dimensional image.

[0091] S3. Use an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and construct a k-nearest neighbor propagation matrix as the adjacency matrix of the graph structure for edge weight propagation;

[0092] S4. Select the most representative image samples for manual annotation, and use the manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure;

[0093] S5. Utilize the edge weights of the graph structure to propagate uncertainty and representativeness of each node to neighboring nodes, and select image samples for active learning using a heuristic sample selection strategy. This process is iterated until the annotation budget is exhausted and active learning ends.

[0094] S6. Calculate training weights for each image sample using the uncertainty and adjacency of the nodes, and retrain the pre-trained autoencoder with the weights to serve as an anomaly detection model.

[0095] S7. Collect image data of the product to be detected and perform the preprocessing described in S1. Finally, input the data into the anomaly detection model and use the reconstruction error to calculate the anomaly score of the product to be detected.

[0096] The method proposed in this embodiment: (1) Updates image annotation information based on the neighbor propagation graph during the active learning phase. Since the relationships between images are modeled when the neighbor graph is created, updating image annotation information is extremely time-efficient, significantly improving its effectiveness in practical tasks.

[0097] (2) After training, the proposed pre-trained autoencoder first obtains image encoding that does not contain low-level features, and further compresses and reconstructs it, which not only obtains a more continuous and smooth low-dimensional representation of the image, but also obtains better anomaly detection performance.

[0098] (3) In the graph active learning stage, the uncertainty and representativeness of the samples are considered simultaneously, and a heuristic sample selection strategy is proposed to select high-value samples. The sample space is fully explored, which effectively improves the performance of active learning.

[0099] Example 2

[0100] This embodiment describes in detail the active learning anomaly detection method based on graph propagation proposed by the present invention.

[0101] First, an automatic optical inspection system is used to randomly collect n unlabeled product images of industrial products to be inspected from real manufacturing scenes. The images are then preprocessed, including template matching, cropping, and normalization, to obtain a product image set, denoted as D. u ={x1,x2,…,x n}, where x i ∈R c×h×w , c, h, and w are the number of channels, height, and width of the image respectively.

[0102] Then, build a pre-trained autoencoder, see Figure 2 The pre-trained autoencoder constructed in this embodiment includes an encoder and a decoder. The encoder includes a pre-trained convolutional neural network and a multi-layer perceptron connected to the pre-trained convolutional neural network. The pre-trained convolutional neural network encodes the input image locally but not including low-level features, and then the multi-layer perceptron performs a low-dimensional representation of the encoded image.

[0103] The decoder consists of a multi-layer perceptron and a deconvolutional neural network connected to the multi-layer perceptron. The low-dimensional representation is input into the multi-layer perceptron of the decoder, which performs a one-time upsampling on it. The upsampling result is input into the deconvolutional neural network, which reconstructs the original image.

[0104] Here, we do not specify which pre-trained convolutional neural network to use, because convolutional neural networks pre-trained with large datasets can obtain better encoding and discard abnormal features, which can meet our needs.

[0105] For the pre-trained autoencoder that has been built, the product image is input into the pre-trained autoencoder and the pre-trained autoencoder is trained. The objective function is expressed as:

[0106]

[0107] in,

[0108]

[0109]

[0110] Among them, J c (θ,γ) represents the reconstruction error; J r (θ,γ) represents the L2 regularization term; D represents the decoder, θ represents the hyperparameter corresponding to the decoder; E represents the encoder, γ represents the hyperparameter corresponding to the encoder; λ is a predefined hyperparameter; x i represents the i-th image in the product image set; |.|_2 represents the L2 norm;

[0111] During the training process, the parameters of the pre-trained convolutional neural network are fixed to ensure that the input obtained from the training part corresponding to the unfixed parameters does not contain low-level features of the image but retains local features. The product image set is divided into batches of fixed size and input into the pre-trained autoencoder. The loss error of the objective function is back-propagated through small batches until the loss error no longer decreases significantly. The training is completed and the autoencoder is recorded as D(E(x i ,γ),θ).

[0112] In this example, Wide ResNet50 was selected as the pre-trained model, and the specific parameters of each layer of the network were determined. The key point of "pre-training autoencoders" is to use pre-trained convolutional neural networks to obtain input that has local characteristics but does not contain local features for the trainable part.

[0113] When training the pre-trained autoencoder, the product image set is divided into batches of fixed size, input into the pre-trained autoencoder, and the loss is calculated (because the minimum value of the objective function is 0, the loss error is the value of the objective function to 0), and then the loss is backpropagated in small batches until the loss no longer decreases significantly, at which point the training converges.

[0114] Assume that the encoding and reconstruction errors of each image in the product image set obtained by the autoencoder are recorded as V = {v1, v2, ..., v n} and ε={e1,e2,…,e n}, v i and e i The calculation expressions are:

[0115] v i =E(x i ,θ)

[0116] t i =|x i -D(v i ,γ)|2

[0117]

[0118] Among them, v i 、ei are the encoding error of the i-th image in the encoder and the reconstruction error of the i-th image in the pre-trained automatic encoder; t i Represents the unnormalized reconstruction error, described as an intermediate variable;

[0119] The k-nearest neighbor propagation matrix is ​​constructed by coding. Let the constructed k-nearest neighbor propagation matrix be expressed as S, and the expression is:

[0120] S=D -1 / 2 W s D -1 / 2

[0121]

[0122] Where D is the degree matrix and the diagonal is the symmetric affine matrix W s The sum of the corresponding rows, other positions are 0; symmetric affine matrix W s The calculation expression is:

[0123]

[0124] Among them, W size is n×n, and the k-nearest neighbor propagation matrix is ​​calculated using the radial kernel function. The k-nearest neighbor distance W between the i-th and j-th samples is ij The calculation expression is:

[0125]

[0126] Among them, knn(i) represents the set of k nearest neighbor samples of sample i.

[0127] The most representative image samples are selected for manual annotation using the spectral clustering method, and the k-nearest neighbor propagation matrix s is used as the input of the spectral clustering method to cluster all image samples into K clusters, denoted as C = {C1, C2, ..., C K}, and select samples with the minimum encoding distance from other samples in each cluster for manual annotation, and the partially annotated image dataset is recorded as D l ={(x1,y1),(x2,y2),…,(x K ,y K )}, where y i ∈{0,1}, where 0 and 1 are normal and abnormal labels respectively; initialize D u The information node f corresponding to each image sample in ui ,f ri ], as nodes of the propagation graph, where:

[0128] f ui =e i

[0129]

[0130] f ui and f ri Represent the uncertainty and representativeness of the i-th sample respectively, and use the information nodes to construct the information matrix recorded as

[0131] The uncertainty and representativeness of each image are propagated through the graph active learning method, and samples are selected for manual annotation through a heuristic sample selection strategy. This is iterated until the annotation budget is exhausted. The specific process is as follows:

[0132] S51. Let the current information matrix be F 0 , the representativeness and uncertainty of the nodes are propagated iteratively through the graph propagation, and the t-th step of the iteration is:

[0133] F t+1 =αSF t +(1-α)F 0 ;

[0134] Until |F t+1 -F t |2<β, where β is a predefined hyperparameter; α=[α1,α2,…,α N ] is used to control the degree of information retention of each node, and the calculation expression is:

[0135]

[0136] Among them, α l and α u are all predefined hyperparameters; let the last step of the iteration be t e , and execute To complete the spread of uncertainty and representativeness;

[0137] S52. Execute heuristic sample uncertainty selection strategy: According to f ui Arrange each unmarked node from high to low, where the first N p The samples corresponding to each node are selected for manual labeling;

[0138] If no abnormal sample is found, the heuristic representative sample selection strategy is executed, and the sample with the largest -f is selected from the unlabeled nodes. ri The samples are manually marked. If abnormal samples are found, or N consecutive p If the heuristic sample selection strategy does not find any abnormal samples, the heuristic sample uncertainty selection strategy is re-executed, and S52 is repeatedly executed until the labeling budget is exhausted.

[0139] After each manual annotation, the node information of the propagation graph is updated using the following formula:

[0140]

[0141] Uncertain sample selection strategy each selects N p After each sample is labeled or each time a sample is selected for labeling by the representative sample selection strategy, S51 is used to propagate the labeling information and update the uncertainty and representativeness of the graph nodes.

[0142] This describes the labeling update and sample selection strategy during active learning. It leverages graph structures to propagate the uncertainty and representativeness of samples. Compared to other active learning strategies, this uses graph propagation to model the relationships between samples before expert annotation. Therefore, even with increasing sample dimensionality, the time required to update labeling information does not increase significantly. Furthermore, this strategy utilizes graph structures to simultaneously propagate both representativeness and uncertainty, which aligns with the prior consistency assumption and has very low computational complexity. This approach effectively avoids the phenomenon of missed clusters common in active anomaly detection while maintaining very low computational time, allowing for a more comprehensive exploration of the entire sample space and achieving better active learning performance.

[0143] A training weight is calculated for each sample through the results of active learning, and the pre-trained autoencoder is retrained as an anomaly detection model based on the weight, and Ω=[ω1,ω2,…,ω N ],ω i ∈R is the training weight, where ω i The calculation expression is:

[0144]

[0145] Among them, F u =[f u1 ,f u2 ,…,f un ];

[0146] The pre-trained autoencoder is retrained using the training weights, and the objective function of the training is recorded as:

[0147]

[0148] in:

[0149]

[0150]

[0151] The training is performed until the objective function converges, and the trained autoencoder is used as the anomaly detection model.

[0152] The product image set that collects image data of the product to be inspected and undergoes the same preprocessing is denoted as x test , abnormal score Score(x test ) is calculated as:

[0153] Score(x test )=|x test -D(E(x test ,γ l ),θ l )|2

[0154] Set an empirical threshold Abnormal score Score(x test ) is greater than The image sample is identified as abnormal and recalled, and the abnormal score Score(x test ) is less than The image samples are considered normal.

[0155] Example 3

[0156] This embodiment is described in conjunction with actual verification results. First, perform the following steps:

[0157] S1. Randomly collect and preprocess n unlabeled images from real scenes using optical sensors and create a dataset;

[0158] S2. Use the dataset to train the pre-trained autoencoder, set the optimizer to Adam, set the batch size of the neural network training to 48, and set the objective function to:

[0159]

[0160] in:

[0161]

[0162]

[0163] Where λ is a predefined hyperparameter, which is set to 5e-5 by default in this embodiment, θ and γ are the hyperparameters of the decoder and encoder respectively. The trained model is denoted as D(E(x i ,γ),θ).

[0164] Here, this embodiment provides an example for the specific parameters of the pre-trained autoencoder, and is not intended to be a limitation on the structure of the pre-trained autoencoder. Specifically, the pre-trained autoencoder is divided into two parts: the encoder and the decoder. The first half of the encoder is a 50-layer wide residual neural network trained using ImageNet. The output of the wide residual neural network is then expanded to 1 dimension for each image, followed by a [2048, esd] linear layer (where 2048 is the input of the linear layer, esd is the output of the linear layer, and is also the dimension of the image output in the encoder), and finally a LeakyReLU nonlinear activation layer; and the first layer of the decoder is a linear layer of size [esd, 128], followed by a LeakyReLU nonlinear activation layer. Activation function, and convert the dimension of each sample to [128,1,1], and then connect 8 deconvolution layers in sequence. The number of output channels, convolution kernel size and deconvolution step size are [128,3,1], [128,3,2], [256,3,2], [128,3,2], [64,3,2], [32,3,2], [32,3,1], [3,3,2] respectively. The first seven deconvolution layers are connected with LeakyReLU nonlinear activation function and batch normalization layer.

[0165] S3. Calculate encoding and reconstruction errors of the image samples using the obtained autoencoder;

[0166] S4. Use the neighbor propagation matrix S as the input of spectral clustering to cluster all samples into K clusters, denoted as C = {C1, C2, ..., C K}, and select samples with the minimum encoding distance from other samples in each cluster to be manually labeled by experts. The labeled image dataset is denoted as D l ={(x1,y1),(x2,y2),…,(x K ,y K )}, where y i ∈{0,1}, where 0 and 1 are normal and abnormal labels respectively.

[0167] S5. Use active learning to select some high-value samples for labeling. Let the maximum budget of active learning single round labeling be N p , the specific steps are:

[0168] 1) Use the approximate solution of diffusion propagation to propagate the uncertainty and representativeness of each sample in the information matrix. Let the current information matrix be F 0 , iteratively propagates the representativeness and uncertainty of the node, and the t-th step of the iteration is:

[0169] F t+1 =αSF t +(1-α)F 0

[0170] Until |F t+1 -F t |2<β, where β is a predefined hyperparameter, which is set to 1e-4 by default in the present invention. α=[α1,α2,…,α N ] is used to control the degree of information retention of each node, which is calculated as follows:

[0171]

[0172] where α l and α u are predefined hyperparameters, which are set to 0.95 and 0.05 by default in this invention. Let the last step of the iteration be t e , and execute To complete the spread of uncertainty and representativeness.

[0173] 2) Execute the heuristic sample uncertainty selection strategy, according to f ui Each unlabeled sample is ranked, with the highest top N p Samples are selected for manual labeling. If no abnormal samples are found, the heuristic representative sample selection strategy is executed to select the sample with the largest 1-f ri samples and manually mark them. If abnormal samples are found, or if N consecutive p If the heuristic sample selection strategy does not find an abnormal sample, the heuristic sample uncertainty selection strategy is re-executed. 2) is iterated until the labeling budget is exhausted.

[0174] S6. Calculate the training weights of the samples and use them to train the pre-trained autoencoder for end-to-end anomaly detection.

[0175] 1) Calculate the training weight of the sample, which is used to retrain the anomaly detection model. Let Ω=[ω1,ω2,…,ω N ],ω i ∈R is the training weight, where ω i The calculation method is:

[0176]

[0177] Among them F u =[f u1 ,f u2 ,…,f un ].

[0178] 2) Use the training weights obtained in 1) to retrain the autoencoder, and its objective function is recorded as:

[0179]

[0180] in:

[0181]

[0182]

[0183] λ is a predefined hyperparameter, which is set to 5e-5 by default in this invention. Finally, the anomaly score is calculated.

[0184] In actual use, an empirical threshold t is set. Samples with anomaly scores greater than t are considered abnormal and recalled, while samples with anomaly scores less than t are considered abnormal and recalled. The sample is considered a normal sample. The present invention recommends setting the threshold to the threshold corresponding to the r%th sample in the test set sorted from largest to smallest, where r% is the acceptable recall cost.

[0185] The present invention randomly collected 6,800 through-hole technology (THT) solder joint images from the production environment of printed circuit board soldering and used them to experiment the effectiveness of the proposed method. A random five-fold partitioning method was used to obtain five sub-datasets, four of which were used as training sets and the remaining one as a test set. A total of five experiments were conducted using a 10% training set as the annotation budget. The performance was evaluated using the image level (AUROC) evaluation method, a mainstream evaluation method for anomaly detection, and the results of the five experiments were displayed as mean and standard deviation. In addition, the experiment also observed the average time taken for the active learning method to update the annotation information once.

[0186] The performance of the proposed method is compared with the state-of-the-art method for semi-supervised anomaly detection PatchCORE, the semi-supervised convolutional autoencoder (CAE) as a baseline, and the latest active anomaly detection method Active Autoencoder (AAE). The results are shown in Table 1.

[0187] Table 1

[0188]

[0189]

[0190] As shown in Table 1, the AUROC of the method proposed in the present invention exceeds the three comparison methods including the most advanced method, which shows the advancement of the method. In addition, AAE updates the annotation information by fine-tuning the neural network, which is also the mainstream method for active anomaly detection in images. However, it shows an update time much higher than the method described above, which reflects that active learning through the propagation of uncertainty and representativeness through graph structure is more practical. Combining the above two points, the anomaly detection method proposed in the present invention propagates the uncertainty and representativeness of the samples through the graph structure, and quickly actively learns and fully explores the sample space through a heuristic sample selection strategy, showing lower active learning update time and stronger active learning performance. In addition, combined with the active learning method of the present invention, the pre-trained convolutional autoencoder also achieved the most advanced semi-supervised anomaly detection performance, which demonstrates the advantage of the present invention in constructing anomaly detection models for automatic optical inspection systems in real production environments.

[0191] Example 4

[0192] like Figure 3 As shown, this embodiment proposes an active learning anomaly detection system based on graph propagation, and the system includes:

[0193] The image acquisition and preprocessing module randomly collects images of several products and performs preprocessing to obtain a product image set;

[0194] The autoencoder construction training module is used to construct a pre-trained autoencoder. The reconstruction error and L2 regularization are used as the objective function. The pre-trained autoencoder is trained using the product image set until convergence. The training is completed and the autoencoder is obtained.

[0195] The graph propagation construction module uses an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and constructs a k-nearest neighbor propagation matrix as the edge weight propagation of the graph structure;

[0196] The information matrix construction module is used to select the most representative image samples for manual annotation, and use manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure;

[0197] The active learning module uses the edge weights of the graph structure to propagate the uncertainty and representativeness of each node, and selects image samples for active learning through a heuristic sample selection strategy until the annotation budget is exhausted and active learning ends;

[0198] The anomaly detection module uses the uncertainty of the nodes and the edge weight to calculate the training weight for each image sample, and retrains the pre-trained autoencoder with the weights to serve as the anomaly detection model;

[0199] The image data of the product to be detected is collected and preprocessed, and finally input into the anomaly detection model, and the anomaly score of the product to be detected is calculated using the reconstruction error.

[0200] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An active learning anomaly detection method based on graph propagation, characterized in that: The method comprises the following steps: S1. Randomly collect images of several products and preprocess them to obtain a product image set; S2. Construct a pre-trained autoencoder. Use the reconstruction error and L2 regularization as the objective function and train the pre-trained autoencoder using the product image set until convergence. Training is complete, and the autoencoder is obtained. S3. Use an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and construct a k-nearest neighbor propagation matrix as the adjacency matrix of the graph structure for edge weight propagation; S4. Select the most representative image samples for manual annotation, and use the manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure; S5. Utilize the edge weights of the graph structure to propagate uncertainty and representativeness of each node to neighboring nodes, and select image samples for active learning using a heuristic sample selection strategy. This process is iterated until the annotation budget is exhausted and active learning ends. S6. Calculate training weights for each image sample using the uncertainty and adjacency of the nodes, and retrain the pre-trained autoencoder with the weights to serve as an anomaly detection model. S7. Collect image data of the product to be detected and perform the preprocessing described in S1. Finally, input the data into the anomaly detection model and use the reconstruction error to calculate the anomaly score of the product to be detected.

2. The active learning anomaly detection method based on graph propagation according to claim 1 is characterized in that In step S1, n unlabeled images are randomly collected from real manufacturing scenes and preprocessed. The preprocessing includes template matching, cropping, and normalization to obtain a product image set, which is recorded as D u ={x1,x2,…,x n }, where x i ∈R c×h×w , x i represents the i-th image in the product image set, c, h, and w are the number of channels, height, and width of the image, respectively.

3. The active learning anomaly detection method based on graph propagation according to claim 1 is characterized in that The pre-trained autoencoder includes an encoder and a decoder. The encoder includes a pre-trained convolutional neural network and a multi-layer perceptron connected to the pre-trained convolutional neural network. The pre-trained convolutional neural network encodes the input image locally but not including low-level features, and the multi-layer perceptron performs a low-dimensional representation of the encoded image. The decoder includes a multi-layer perceptron and a deconvolutional neural network connected to the multi-layer perceptron. The low-dimensional representation is input into the multi-layer perceptron of the decoder, and the multi-layer perceptron performs a one-time upsampling on it. The upsampling result is input into the deconvolutional neural network, and the deconvolutional neural network reconstructs the original image.

4. The active learning anomaly detection method based on graph propagation according to claim 1 is characterized in that In step S2, the objective function is expressed as: in, Among them, J c (θ,γ) represents the reconstruction error; J r (θ,γ) represents the L2 regularization term; D represents the decoder, θ represents the hyperparameter corresponding to the decoder; E represents the encoder, γ represents the hyperparameter corresponding to the encoder; λ is a predefined hyperparameter; x i represents the i-th image in the product image set; |.|2 represents the L2 norm; During the training process, the parameters of the pre-trained convolutional neural network are fixed to ensure that the input obtained from the training part corresponding to the unfixed parameters does not contain low-level features of the image but retains local features. The product image set is divided into batches of fixed size and input into the pre-trained autoencoder. The loss error of the objective function is back-propagated through small batches until the loss error no longer decreases significantly. The training is completed and the autoencoder is recorded as D(E(x i ,γ),θ).

5. The active learning anomaly detection method based on graph propagation according to claim 4 is characterized in that In step S3, the encoding and reconstruction errors of each image in the product image set obtained by the autoencoder are recorded as V = {v1, v2, ..., v n } and ε={e1,e2,…,e n }, v i and e i The calculation expressions are: v i =E(x i ,i) t i =|x i -D(v i ,γ)|2 Among them, v i 、e i are the encoding error of the i-th image in the encoder and the reconstruction error of the i-th image in the pre-trained automatic encoder; t i Represents the unnormalized reconstruction error, described as an intermediate variable; Let the constructed k-nearest neighbor propagation matrix be expressed as S, which is expressed as: S=D -1 / 2 W s D -1 / 2 Where D is the degree matrix and the diagonal is the symmetric affine matrix W s The sum of the corresponding rows, other positions are 0; symmetric affine matrix W s The calculation expression is: Among them, W size is n×n, and the k-nearest neighbor propagation matrix is ​​calculated using the radial kernel function. The k-nearest neighbor distance W between the i-th and j-th samples is ij The calculation expression is: Among them, knn(i) represents the set of k nearest neighbor samples of sample i.

6. The active learning anomaly detection method based on graph propagation according to claim 5 is characterized in that In step S4, the most representative image samples are selected for manual annotation using the spectral clustering method, and the k-nearest neighbor propagation matrix is ​​used as the input of the spectral clustering method to cluster all image samples into K clusters, denoted as C = {C1, C2, ..., C K }, and select samples with the minimum encoding distance from other samples in each cluster for manual annotation, and the partially annotated image dataset is recorded as D l ={(x1,y1),(x2,y2),…,(x K ,y K )}, where y i ∈{0,1}, where 0 and 1 are normal and abnormal labels respectively; initialize D u The information node f corresponding to each image sample in ui ,f ri ], as a node of the propagation graph, D u Represents a set of product images, where: f ui =e i f ui and f ri Represent the uncertainty and representativeness of the i-th sample respectively, and use the information nodes to construct the information matrix recorded as 7. The active learning anomaly detection method based on graph propagation according to claim 6 is characterized in that The specific process of step S5 is: S51. Let the current information matrix be F 0 , the representativeness and uncertainty of the nodes are propagated iteratively through the graph propagation, and the t-th step of the iteration is: F t+1 =αSF t +(1-α)F 0 ; Until |F t+1 -F t |2<β, where β is a predefined hyperparameter; α=[α1,α2,…,α N ] is used to control the degree of information retention of each node, and the calculation expression is: Among them, α l and α u are all predefined hyperparameters; let the last step of the iteration be t e , and execute To complete the spread of uncertainty and representativeness; S52. Execute heuristic sample uncertainty selection strategy: According to f ui Arrange each unmarked node from high to low, where the first N p The samples corresponding to each node are selected for manual labeling; If no abnormal sample is found, the heuristic representative sample selection strategy is executed, and the sample with the largest -f is selected from the unlabeled nodes. ri The samples are manually marked. If abnormal samples are found, or N consecutive p If the heuristic sample selection strategy does not find any abnormal samples, the heuristic sample uncertainty selection strategy is re-executed, and S52 is repeatedly executed until the labeling budget is exhausted; After each manual annotation, the node information of the propagation graph is updated using the following formula: Uncertain sample selection strategy each selects N p After each sample is labeled or each time a sample is selected for labeling by the representative sample selection strategy, S51 is used to propagate the labeling information and update the uncertainty and representativeness of the graph nodes.

8. The active learning anomaly detection method based on graph propagation according to claim 7 is characterized in that Let Ω=[ω1,ω2,…,ω N ],ω i ∈R is the training weight, where ω i The calculation expression is: Among them, F u =[f u1 ,f u2 ,…,f un ]; The pre-trained autoencoder is retrained using the training weights, and the objective function of the training is recorded as: in: The training is performed until the objective function converges, and the trained autoencoder is used as the anomaly detection model.

9. The active learning anomaly detection method based on graph propagation according to claim 8, characterized in that The product image set that collects image data of the product to be inspected and performs the preprocessing described in S1 is denoted as x test , abnormal score Score(x test ) is calculated as: Score(x test )=|x test -D(E(x test ,c l ),θ l )|2 Set an empirical threshold t, anomaly score Score(x test ) is greater than t, the image sample is identified as abnormal and recalled, and the abnormal score Score(x test ) is less than t and is considered normal.

10. An active learning anomaly detection system based on graph propagation, characterized in that The system comprises: The image acquisition and preprocessing module randomly collects images of several products and performs preprocessing to obtain a product image set; The autoencoder construction training module is used to construct a pre-trained autoencoder. The reconstruction error and L2 regularization are used as the objective function. The pre-trained autoencoder is trained using the product image set until convergence. The training is completed and the autoencoder is obtained. The graph propagation construction module uses an autoencoder to obtain the encoding and reconstruction errors of each image in the product image set, and constructs a k-nearest neighbor propagation matrix as the edge weight propagation of the graph structure; The information matrix construction module is used to select the most representative image samples for manual annotation, and use manual annotation and reconstruction errors to construct an information matrix containing sample uncertainty and representativeness, which is used to initialize the node information of the graph structure; The active learning module uses the edge weights of the graph structure to propagate the uncertainty and representativeness of each node, and selects image samples for active learning through a heuristic sample selection strategy until the annotation budget is exhausted and active learning ends; The anomaly detection module uses the uncertainty of the nodes and the edge weight to calculate the training weight for each image sample, and retrains the pre-trained autoencoder with the weights to serve as the anomaly detection model; The image data of the product to be detected is collected and preprocessed, and finally input into the anomaly detection model, and the anomaly score of the product to be detected is calculated using the reconstruction error.

Images