Semi-supervised anomaly detection method based on memory enhancement and pseudo marking

Abstract

The invention relates to a semi-supervised anomaly detection method based on memory enhancement and pseudo marking, belongs to the technical field of machine vision and anomaly detection, and solves the problems of low anomaly detection accuracy, poor model stability and high data marking cost in the prior art. The method comprises the following steps: inputting image data into a pre-constructed QC-Net model to obtain an anomaly detection result; the QC-Net model comprises an input module, an Encoder-Decoder module, a VAE-GAN encoder, a pseudo tagging device, a memory module and an anomaly detection module; the Encoder-Decoder module is used for carrying out feature extraction on the image data of the product needing to be detected and generating image reconstruction data; the VAE-GAN encoder generates potential abnormal information based on the image data of the product to be detected; the pseudo tag device generates an abnormal tag based on the features extracted by the Encoder-Decoder module; the memory module stores historical abnormal image data and corresponding abnormal detection results; and the anomaly detection module obtains an anomaly detection result based on the image reconstruction data, the potential anomaly information, the anomaly reconstruction data, the anomaly label and the historical anomaly image data.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision and anomaly detection, and in particular to a semi-supervised anomaly detection method based on memory enhancement and pseudo-labeling. Background Art

[0002] With the development of intelligent industrial production, the demand for automated product quality inspection is growing. In product inspection, image-based anomaly detection has become a hot topic in research and application due to its intuitive and efficient characteristics. By analyzing product images, it is possible to quickly and accurately identify whether the product has anomalies, which helps to improve production efficiency and product quality.

[0003] However, when faced with complex image data, existing anomaly detection methods lack effective feature extraction and fusion mechanisms, making it difficult to fully and accurately capture abnormal features in images, resulting in low accuracy of anomaly detection and failure to meet the needs of high-precision anomaly detection in actual production. In the detection of surface defects of industrial products, complex textures, lighting and other factors will interfere with the identification of anomalies, resulting in many misjudgments and missed detections in the detection results.

[0004] Traditional supervised learning anomaly detection methods require a large amount of labeled data to train the model, but in actual applications, obtaining large-scale labeled data is often costly, time-consuming and laborious. In addition, for some rare anomalies, the scarcity of labeled data and the lack of sufficient samples limit the application effect of supervised learning methods in anomaly detection. When facing complex image data, the existing semi-supervised anomaly detection methods lack effective feature extraction and fusion mechanisms, as well as the rational use of historical anomaly information, resulting in insufficient accuracy and stability of anomaly detection, which cannot meet the needs of high-precision anomaly detection in actual production. When processing different batches of data or in complex and changing environments, the stability of its detection results is poor and the fluctuations are large. It cannot provide reliable and consistent detection results, and is difficult to be competent for some application scenarios with high stability requirements. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a semi-supervised anomaly detection method based on memory enhancement and pseudo-labeling, so as to solve the problems of low accuracy, poor model stability and high data annotation cost of existing anomaly detection.

[0006] On the one hand, an embodiment of the present invention provides a semi-supervised anomaly detection method based on memory enhancement and pseudo-labeling, comprising the following steps:

[0007] Obtain image data of the product to be inspected, and input the image data of the product to be inspected into the pre-built QC-Net model to obtain anomaly detection results; wherein the QC-Net model includes an input module, an Encoder-Decoder module, a VAE-GAN encoder, a pseudo labeler, a memory module, and anomaly detection module;

[0008] An input module is used to receive image data of the product to be detected, and send the image data of the product to be detected to the Encoder-Decoder module, the VAE-GAN encoder, the pseudo labeler, the anomaly detection module and the memory module respectively;

[0009] The Encoder-Decoder module extracts features from the image data of the product to be inspected and sends them to the pseudo labeler; and generates image reconstruction data based on the extracted features and sends the image reconstruction data to the anomaly detection module;

[0010] A VAE-GAN encoder, used for generating potential anomaly information based on the image data of the product to be detected, generating anomaly reconstruction data based on the potential anomaly information, and sending the potential anomaly information and the anomaly reconstruction data to the anomaly detection module;

[0011] Pseudo labeler, used to generate a label based on the features extracted by the Encoder-Decoder module and send it to the anomaly detection module;

[0012] A memory module, used to store historical abnormal image data and its corresponding abnormal detection results;

[0013] The anomaly detection module obtains an anomaly detection result based on the image data of the product to be detected, the image reconstruction data, potential anomaly information, anomaly reconstruction data, pseudo labels, historical anomaly image data stored in the memory module and its corresponding anomaly detection results, and transmits the anomaly detection result to the memory module.

[0014] As a further improvement of the present application, the Encoder-Decoder module includes an Encoder unit and a Decoder unit;

[0015] The encoder unit includes a first convolutional layer, a first pooling layer, a second convolutional layer, and a second pooling layer connected end to end in sequence;

[0016] The first convolution layer extracts the features of the image data, and the first pooling layer performs a pooling operation on the features extracted by the first convolution layer. The second convolution layer performs a convolution operation on the output of the first pooling layer to extract the features of the image data. The second pooling layer performs dimensionality reduction processing on the output of the second convolution layer to obtain the final features of the image data. The output of the second pooling layer is used as the output of the Encoder unit and connected to the input of the Decoder unit and the pseudo labeler.

[0017] The decoder unit includes a transposed convolution layer, an upsampling layer, and a decoder convolution layer connected end to end;

[0018] The transposed convolution layer receives the output of the second pooling layer and performs a transposed convolution operation on the final features of the image data; the upsampling layer upsamples the output of the transposed convolution layer to increase the feature space dimension; the decoder convolution layer performs a convolution operation on the output of the upsampling layer to generate image reconstruction data. The output of the decoder convolution layer is used as the output of the decoder unit and connected to the input of the anomaly detection module.

[0019] As a further improvement of the present application, the VAE-GAN encoder includes an encoder and a decoder connected end to end, the encoder is used to generate potential abnormality information based on the image data; the decoder is used to generate reconstructed image data based on the potential abnormality information.

[0020] As a further improvement of the present application, the encoder includes a first convolution layer, a first activation function layer, a first pooling layer, a second convolution layer, a second activation function layer, a second pooling layer, a Flatten layer, a first fully connected layer, a third activation function layer, and a second fully connected layer connected end to end in sequence;

[0021] Among them, the first convolution layer extracts the features of the image data, the first activation function layer performs nonlinear processing on the output of the first convolution layer; the first pooling layer performs a pooling operation on the output of the first encoder activation function layer to reduce the feature dimension after nonlinear processing; the second convolution layer performs a convolution operation on the output of the first pooling layer to extract the features of the image data; the second activation function layer performs nonlinear processing on the output of the second convolution layer; the second pooling layer performs a pooling operation on the output of the second activation function layer; the Flatten layer converts the output of the second pooling layer into a one-dimensional feature vector; the first fully connected layer performs feature compression on the one-dimensional feature vector; the third activation function layer performs nonlinear processing on the output of the first fully connected layer; the second fully connected layer performs feature mapping on the output of the third activation function layer to generate potential abnormal information of the image data.

[0022] As a further improvement of the present application, the decoder includes a first transposed convolution layer, a first activation function layer, an upsampling layer, a second transposed convolution layer, a second activation function layer, and a decoder convolution layer connected end to end in sequence;

[0023] Among them, the first transposed convolution layer performs a transposed convolution operation on the potential abnormal information received from the encoder; the first activation function layer performs nonlinear processing on the output of the first transposed convolution layer; the upsampling layer upsamples the output of the first activation function layer; the second transposed convolution layer performs a transposed convolution operation on the output of the upsampling layer; the second activation function layer performs nonlinear processing on the output of the second transposed convolution layer; the decoder convolution layer performs a convolution operation on the output of the second activation function layer to obtain reconstructed image data.

[0024] As a further improvement of the present application, the pseudo labeler includes an input layer, a fully connected layer, an activation function layer, and an output layer connected end to end in sequence;

[0025] The input layer receives the final features of the image data output by the Encoder unit and transmits them to the fully connected layer;

[0026] The fully connected layer performs spatial mapping on the final features of the image data to generate the number of feature categories;

[0027] An activation function layer generates a distribution probability of each feature category based on the number of feature categories through a Softmax function;

[0028] The output layer generates anomaly labels based on the distribution probability of each feature category and transmits them to the anomaly detection module.

[0029] As a further improvement of the present application, the anomaly detection module includes:

[0030] A reconstruction deviation calculation unit, used for calculating the reconstruction deviation based on the image reconstruction data and the image data of the product to be inspected;

[0031] An abnormal deviation calculation unit, used for calculating the abnormal deviation based on the abnormal reconstruction data and the image data of the product to be detected;

[0032] A pseudo-label deviation calculation unit, used for calculating the pseudo-label deviation based on the abnormal label and the real label of the pre-labeled image data of the product to be detected;

[0033] A regularization deviation calculation unit, used for calculating a regularization deviation based on potential abnormal information;

[0034] The abnormality positioning unit is used to fuse the reconstruction deviation, the abnormal deviation, the pseudo-label deviation and the regularization deviation to obtain a comprehensive deviation feature; retrieve the historical abnormal image with the highest similarity to the comprehensive deviation feature in the memory module, and determine the position of the abnormal area in the image of the product to be inspected based on the abnormal area of ​​the historical abnormal image;

[0035] The anomaly classification unit is used to obtain the anomaly category based on the anomaly label output by the pseudo labeler.

[0036] As a further improvement of the present application, the reconstruction deviation is calculated based on the image reconstruction data and the image data of the product to be inspected as shown in calculation formula (1);

[0037]

[0038] Among them, n is the number of image data of the product to be detected, y i is the image data of the i-th product to be inspected, Reconstruct data for the i-th image.

[0039] The calculation of the abnormal deviation based on the abnormal reconstruction data and the image data of the product to be inspected is shown in calculation formula (2);

[0040]

[0041] Among them, n is the number of image data of the product to be detected, a i is the image data of the i-th product to be inspected, Reconstruct the data for the i-th anomaly.

[0042] As a further improvement of the present application, the pseudo label deviation is calculated based on the pseudo label and the real label of the pre-labeled image data of the product to be detected as shown in formula (3);

[0043]

[0044] Among them, n is the number of image data of the product to be detected, y ic is the encoding of the true label of the i-th image data, Encode the pseudo label of the i-th image data.

[0045] As a further improvement of the present application, the regularization deviation is calculated based on the potential abnormal information as shown in formula (4);

[0046] L latent =D KL (q(z|xi)||p(z)) (4)

[0047] Among them, D KL is the KL divergence, q(z|xi) is the potential abnormal information, and p(z) is the prior distribution of the abnormal information.

[0048] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0049] 1. The present invention constructs a QC-Net model, including an Encoder-Decoder module, a VAE-GAN encoder, and a pseudo labeler. Each module extracts, analyzes, and reconstructs features of image data from different angles, so as to capture abnormal features more comprehensively and accurately, calculate deviations based on multiple information, perform feature fusion, and comprehensively judge abnormal situations, thereby improving the accuracy of abnormality detection in complex images and solving the problem of low accuracy of abnormality detection in complex images in existing methods.

[0050] 2. The memory module of the present invention can store historical abnormal image data and its corresponding abnormal detection results. When detecting new product images, the abnormal location unit retrieves the historical abnormal image with the highest similarity to the comprehensive deviation feature in the memory module, and determines the location of the abnormal area based on historical experience. The model can make full use of historical information, reduce the volatility of detection results, and enhance stability under complex data, thus solving the problem of insufficient stability of existing models.

[0051] 3. The present invention adopts a semi-supervised learning approach, and generates pseudo labels based on a small amount of labeled data through a pseudo labeler. When calculating the pseudo label deviation, the relationship between the pseudo label and other features is fully considered, so that the model can more accurately utilize the information in the unlabeled data, effectively improving the accuracy of semi-supervised anomaly detection, reducing the demand for large-scale labeled data, reducing the cost of data labeling, and improving the practicability and scalability of the detection method.

[0052] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0054] Figure 1 A schematic diagram of the structure of a QC-Net model provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0056] A specific embodiment of the present invention discloses a semi-supervised anomaly detection method based on memory enhancement and pseudo-labeling, comprising:

[0057] Obtain image data of the product to be inspected, and input the image data of the product to be inspected into the pre-built QC-Net model to obtain anomaly detection results; wherein the QC-Net model includes an input module, an Encoder-Decoder module, a VAE-GAN encoder, a pseudo labeler, a memory module, and anomaly detection module;

[0058] An input module is used to receive image data of the product to be detected, and send the image data of the product to be detected to the Encoder-Decoder module, the VAE-GAN encoder, the pseudo labeler, the anomaly detection module and the memory module respectively;

[0059] The Encoder-Decoder module extracts features from the image data of the product to be inspected and sends them to the pseudo labeler; and generates image reconstruction data based on the extracted features and sends the image reconstruction data to the anomaly detection module;

[0060] A VAE-GAN encoder, used for generating potential anomaly information based on the image data of the product to be detected, generating anomaly reconstruction data based on the potential anomaly information, and sending the potential anomaly information and the anomaly reconstruction data to the anomaly detection module;

[0061] Pseudo labeler, used to generate a label based on the features extracted by the Encoder-Decoder module and send it to the anomaly detection module;

[0062] A memory module, used to store historical abnormal image data and its corresponding abnormal detection results;

[0063] The anomaly detection module obtains an anomaly detection result based on the image data of the product to be detected, the image reconstruction data, potential anomaly information, anomaly reconstruction data, pseudo labels, historical anomaly image data stored in the memory module and its corresponding anomaly detection results, and transmits the anomaly detection result to the memory module.

[0064] like Figure 1 As shown in the figure, the image data of the product to be inspected is inspected through the pre-built QC-Net model. The QC-Net model includes an input module, an Encoder-Decoder module, a VAE-GAN encoder, a pseudo labeler, a memory module, and an anomaly detection module.

[0065] The input module is the input end of the QC-Net model and receives the image data of the product to be tested. The image data of the product to be tested refers to a two-dimensional or three-dimensional image including the appearance, structure and other information of the product to be tested, which can be a grayscale image, a color image or other forms of image data. The input module sends the received image data to the Encoder-Decoder module, the VAE-GAN encoder, the anomaly detection module and the memory module respectively.

[0066] The Encoder-Decoder module extracts features from the image data of the product to be inspected and sends them to the pseudo labeler; it also generates image reconstruction data based on the extracted features and sends the image reconstruction data to the anomaly detection module. The extracted features include features that can represent image information, such as edges, textures, and shapes. Generating image reconstruction data means restoring the extracted features to an image. Through the combination of the transposed convolution layer, the upsampling layer, and the convolution layer, the low-dimensional features are gradually restored to a spatial size similar to the original image. The difference between the original image and the reconstructed image can be judged to detect anomalies.

[0067] Specifically, the Encoder-Decoder module includes an Encoder unit and a Decoder unit;

[0068] The encoder unit includes a first convolutional layer, a first pooling layer, a second convolutional layer, and a second pooling layer connected end to end in sequence;

[0069] The first convolution layer extracts the features of the image data, and the first pooling layer performs a pooling operation on the features extracted by the first convolution layer. The second convolution layer performs a convolution operation on the output of the first pooling layer to extract the features of the image data. The second pooling layer performs dimensionality reduction processing on the output of the second convolution layer to obtain the final features of the image data. The output of the second pooling layer is used as the output of the Encoder unit and connected to the input of the Decoder unit and the pseudo labeler.

[0070] The decoder unit includes a transposed convolution layer, an upsampling layer, and a decoder convolution layer connected end to end;

[0071] The transposed convolution layer receives the output of the second pooling layer and performs a transposed convolution operation on the final features of the image data; the upsampling layer upsamples the output of the transposed convolution layer to increase the feature space dimension; the decoder convolution layer performs a convolution operation on the output of the upsampling layer to generate image reconstruction data. The output of the decoder convolution layer is used as the output of the decoder unit and connected to the input of the anomaly detection module.

[0072] The VAE-GAN encoder is used to generate potential abnormality information based on the image data of the product to be detected, generate abnormal reconstruction data based on the potential abnormality information, and send the potential abnormality information and the abnormal reconstruction data to the abnormality detection module.

[0073] Specifically, the VAE-GAN encoder includes an encoder and a decoder connected end to end. The encoder is used to generate potential anomaly information based on image data; the decoder is used to generate reconstructed image data based on the potential anomaly information. Potential anomaly information is a potential representation obtained by encoding image data, which can contain abnormal features existing in the image. The potential anomaly information needs to be restored to observable image information by the decoder. The abnormal reconstructed data is an image obtained by decoding the potential anomaly information through the decoder part of the VAE-GAN encoder. The decoded image will show obvious differences in the abnormal area compared with the original image. This difference can be used as a basis for anomaly detection.

[0074] Based on the potential anomaly information, anomaly reconstruction data is generated, which provides another angle for detecting anomalies. The potential anomaly information and anomaly reconstruction data are sent to the anomaly detection module.

[0075] The encoder includes a first convolution layer, a first activation function layer, a first pooling layer, a second convolution layer, a second activation function layer, a second pooling layer, a Flatten layer, a first fully connected layer, a third activation function layer, and a second fully connected layer, which are connected end to end in sequence; wherein the first convolution layer extracts features of image data, and the first activation function layer performs nonlinear processing on the output of the first convolution layer; the first pooling layer performs a pooling operation on the output of the first encoder activation function layer to reduce the feature dimension after nonlinear processing; the second convolution layer performs a convolution operation on the output of the first pooling layer to extract features of image data; the second activation function layer performs nonlinear processing on the output of the second convolution layer; the second pooling layer performs a pooling operation on the output of the second activation function layer; the Flatten layer converts the output of the second pooling layer into a one-dimensional feature vector; the first fully connected layer performs feature compression on the one-dimensional feature vector; the third activation function layer performs nonlinear processing on the output of the first fully connected layer; the second fully connected layer performs feature mapping on the output of the third activation function layer to generate potential abnormal information of the image data.

[0076] The decoder includes a first transposed convolution layer, a first activation function layer, an upsampling layer, a second transposed convolution layer, a second activation function layer, and a decoder convolution layer which are connected end to end in sequence;

[0077] Among them, the first transposed convolution layer performs a transposed convolution operation on the potential abnormal information received from the encoder; the first activation function layer performs nonlinear processing on the output of the first transposed convolution layer; the upsampling layer upsamples the output of the first activation function layer; the second transposed convolution layer performs a transposed convolution operation on the output of the upsampling layer; the second activation function layer performs nonlinear processing on the output of the second transposed convolution layer; the decoder convolution layer performs a convolution operation on the output of the second activation function layer to obtain abnormal reconstructed image data.

[0078] The pseudo labeler is used to generate anomaly labels based on the features extracted by the Encoder-Decoder module and send them to the anomaly detection module. The pseudo label is a predicted label obtained by processing the input layer, the fully connected layer, the activation function layer, and the output layer. It can provide additional supervision information when the labeled data is limited. For product images, the pseudo labeler can generate anomaly labels based on the distribution probability of each feature category.

[0079] Specifically, the pseudo labeler includes an input layer, a fully connected layer, an activation function layer, and an output layer which are sequentially connected end to end;

[0080] The input layer receives the final features of the image data output by the Encoder unit and transmits them to the fully connected layer;

[0081] The fully connected layer performs spatial mapping on the final features of the image data to generate the number of feature categories;

[0082] An activation function layer generates a distribution probability of each feature category based on the number of feature categories through a Softmax function;

[0083] The output layer generates anomaly labels based on the distribution probability of each feature category and transmits them to the anomaly detection module.

[0084] Furthermore, the abnormality detection module obtains an abnormality detection result based on the image data of the product to be detected, the image reconstruction data, the potential abnormality information, the abnormal reconstruction data, the abnormality label, the historical abnormal image data stored in the memory module and the corresponding abnormality detection result, and transmits the abnormality detection result to the memory module. The abnormality detection result includes an abnormal area and an abnormality type.

[0085] The anomaly detection module includes:

[0086] The reconstruction deviation calculation unit is used to calculate the reconstruction deviation based on the image reconstruction data and the image data of the product to be inspected. The reconstruction deviation calculation unit calculates the reconstruction deviation by comparing the difference between the image reconstruction data and the original image data.

[0087] The reconstruction deviation is calculated based on the image reconstruction data and the image data of the product to be inspected as shown in formula (1);

[0088]

[0089] Among them, n is the number of image data of the product to be detected, y i is the image data of the i-th product to be inspected, Reconstruct data for the i-th image.

[0090] The abnormal deviation calculation unit is used to calculate the abnormal deviation based on the abnormal reconstruction data and the image data of the product to be detected. The abnormal deviation calculation unit calculates the abnormal deviation by comparing the difference between the abnormal reconstruction data and the image data of the product to be detected.

[0091] The abnormal deviation is calculated based on the abnormal reconstruction data and the image data of the product to be inspected as shown in formula (2);

[0092]

[0093] Among them, n is the number of image data of the product to be detected, a i is the image data of the i-th product to be inspected, Reconstruct the data for the i-th anomaly.

[0094] The pseudo label deviation calculation unit is used to calculate the pseudo label deviation based on the abnormal label and the real label of the image data of the product to be detected. The pseudo label deviation calculation unit calculates the pseudo label deviation by comparing the difference between the abnormal label generated by the pseudo labeler and the pre-labeled real label.

[0095] The pseudo-label deviation is calculated based on the abnormal label and the true label of the pre-labeled image data of the product to be inspected as shown in formula (3);

[0096]

[0097] Among them, n is the number of image data of the product to be detected, y ic is the encoding of the true label of the i-th image data, Encode the pseudo label of the i-th image data.

[0098] The regularization deviation calculation unit is used to calculate the regularization deviation based on the potential abnormal information. The regularization deviation measures the difference between the distribution of the potential abnormal information and the prior distribution through the divergence. By calculating the divergence, it can be known whether the distribution of the potential abnormal information generated by the current model conforms to the expected prior distribution. Divergence is a constraint condition of the model. By minimizing the divergence, the model can be trained to make the distribution of the potential abnormal information closer to the prior distribution, making the model prediction more accurate.

[0099] The regularization deviation is calculated based on the potential abnormal information as shown in formula (4);

[0100] L latent =D KL (q(z|xi)||p(z)) (4)

[0101] Among them, D KL is the KL divergence, q(z|xi) is the potential abnormal information, and p(z) is the prior distribution of the abnormal information.

[0102] The anomaly localization unit is used to perform feature fusion on the reconstruction deviation, the anomaly deviation, the pseudo-label deviation and the regularization deviation to obtain a comprehensive deviation feature; retrieve the historical anomaly image with the highest similarity to the comprehensive deviation feature in the memory module, and determine the position of the abnormal area in the image of the product to be inspected based on the abnormal area of ​​the historical anomaly image.

[0103] The anomaly localization unit fuses the reconstruction deviation, anomaly deviation, pseudo label deviation and regularization deviation to obtain the comprehensive deviation feature. The comprehensive deviation feature is shown in formula (5).

[0104] L total =ω1.L recon +ω2.L anomaly +ω3.L latent +ω4.L pseudo (5);

[0105] Among them, L total is the comprehensive deviation characteristic; L recon is the reconstruction deviation; L anomaly is abnormal deviation; L latent is the pseudo label deviation; L pseudo is the regularization bias. ω1, ω2, ω3, and ω4 are the weights of reconstruction bias, anomaly bias, pseudo-label bias, and regularization bias, which are pre-set based on different anomaly detection tasks through experiments or experience. If the reconstructed image is significantly different from the original image, the weight of the reconstruction bias can be increased; for tasks that rely more on the accuracy of pseudo-labels, the weight of the pseudo-label bias can be increased.

[0106] When retrieving the historical abnormal image with the highest similarity to the comprehensive deviation feature in the memory module, the historical abnormal image with the highest similarity to the comprehensive deviation feature can be retrieved in the memory module by using the Euclidean distance or the hash value as the similarity metric. The abnormal classification unit maps the abnormal label to a specific abnormal category according to the abnormal label output by the pseudo labeler.

[0107] Furthermore, the memory module is used to store historical abnormal image data and its corresponding abnormal detection results. The historical abnormal image data is image data that has been detected before and is determined to have abnormalities, and its corresponding abnormal detection results include abnormal areas and abnormal types.

[0108] The above-mentioned embodiments of the present invention have the following beneficial effects: by constructing a QC-Net model, including an Encoder-Decoder module, a VAE-GAN encoder, and a pseudo labeler, each module extracts, analyzes, and reconstructs features of image data from different angles, can more comprehensively and accurately capture abnormal features, calculate deviations based on multiple information and perform feature fusion, comprehensively judge abnormal situations, improve the accuracy of complex image anomaly detection, and solve the problem of low accuracy of complex image anomaly detection in existing methods; the memory module can store historical abnormal image data and its corresponding abnormal detection results. When detecting new product images, the abnormal location unit detects and locates the abnormal image in the memory module. The historical abnormal images with the highest similarity to the comprehensive deviation features are retrieved, and the location of the abnormal area is determined based on historical experience. The model can make full use of historical information, reduce the volatility of the detection results, enhance the stability under complex data, and solve the problem of insufficient stability of the existing model. A semi-supervised learning method is adopted to generate pseudo labels based on a small amount of labeled data through a pseudo labeler. When calculating the pseudo label deviation, the relationship between the pseudo label and other features is fully considered, so that the model can more accurately utilize the information in the unlabeled data, effectively improving the accuracy of semi-supervised anomaly detection, reducing the demand for large-scale labeled data, reducing the cost of data labeling, and improving the practicality and scalability of the detection method.

[0109] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0110] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A semi-supervised anomaly detection method based on memory enhancement and pseudo-labeling, comprising the following steps: Obtain image data of the product to be inspected, and input the image data of the product to be inspected into the pre-built QC-Net model to obtain anomaly detection results; wherein the QC-Net model includes an input module, an Encoder-Decoder module, a VAE-GAN encoder, a pseudo labeler, a memory module, and anomaly detection module; An input module is used to receive image data of the product to be detected, and send the image data of the product to be detected to the Encoder-Decoder module, the VAE-GAN encoder, the pseudo labeler, the anomaly detection module and the memory module respectively; The Encoder-Decoder module extracts features from the image data of the product to be inspected and sends them to the pseudo labeler; and generates image reconstruction data based on the extracted features and sends the image reconstruction data to the anomaly detection module; A VAE-GAN encoder, used for generating potential anomaly information based on the image data of the product to be detected, generating anomaly reconstruction data based on the potential anomaly information, and sending the potential anomaly information and the anomaly reconstruction data to the anomaly detection module; Pseudo labeler, used to generate anomaly labels based on the features extracted by the Encoder-Decoder module and send them to the anomaly detection module; A memory module, used to store historical abnormal image data and its corresponding abnormal detection results; The abnormality detection module obtains an abnormality detection result based on the image data of the product to be detected, the image reconstruction data, potential abnormality information, abnormal reconstruction data, abnormal labels, historical abnormal image data stored in the memory module and the corresponding abnormality detection results, and transmits the abnormality detection result to the memory module.

2. The method according to claim 1, characterized in that The Encoder-Decoder module includes an Encoder unit and a Decoder unit; The encoder unit includes a first convolutional layer, a first pooling layer, a second convolutional layer, and a second pooling layer connected end to end in sequence; The first convolution layer extracts the features of the image data, and the first pooling layer performs a pooling operation on the features extracted by the first convolution layer. The second convolution layer performs a convolution operation on the output of the first pooling layer to extract the features of the image data. The second pooling layer performs dimensionality reduction processing on the output of the second convolution layer to obtain the final features of the image data. The output of the second pooling layer is used as the output of the Encoder unit and connected to the input of the Decoder unit and the pseudo labeler. The decoder unit includes a transposed convolution layer, an upsampling layer, and a decoder convolution layer connected end to end; The transposed convolution layer receives the output of the second pooling layer and performs a transposed convolution operation on the final features of the image data; the upsampling layer upsamples the output of the transposed convolution layer to increase the feature space dimension; The decoder convolution layer performs a convolution operation on the output of the upsampling layer to generate image reconstruction data. The output of the decoder convolution layer is used as the output of the decoder unit and connected to the input of the anomaly detection module.

3. The method according to claim 1, characterized in that: The VAE-GAN encoder includes an encoder and a decoder connected end to end. The encoder is used to generate potential abnormal information based on image data; A decoder is used to generate reconstructed image data based on the potential anomaly information.

4. The method according to claim 3, characterized in that The encoder includes a first convolution layer, a first activation function layer, a first pooling layer, a second convolution layer, a second activation function layer, a second pooling layer, a Flatten layer, a first fully connected layer, a third activation function layer, and a second fully connected layer, which are connected end to end in sequence; Among them, the first convolution layer extracts the features of the image data, the first activation function layer performs nonlinear processing on the output of the first convolution layer; the first pooling layer performs a pooling operation on the output of the first encoder activation function layer to reduce the feature dimension after nonlinear processing; the second convolution layer performs a convolution operation on the output of the first pooling layer to extract the features of the image data; the second activation function layer performs nonlinear processing on the output of the second convolution layer; the second pooling layer performs a pooling operation on the output of the second activation function layer; the Flatten layer converts the output of the second pooling layer into a one-dimensional feature vector; the first fully connected layer performs feature compression on the one-dimensional feature vector; the third activation function layer performs nonlinear processing on the output of the first fully connected layer; the second fully connected layer performs feature mapping on the output of the third activation function layer to generate potential abnormal information of the image data.

5. The method according to claim 4, characterized in that The decoder comprises a first transposed convolution layer, a first activation function layer, an upsampling layer, a second transposed convolution layer, a second activation function layer, and a decoder convolution layer which are sequentially connected end to end; Among them, the first transposed convolution layer performs a transposed convolution operation on the potential abnormal information received from the encoder; the first activation function layer performs nonlinear processing on the output of the first transposed convolution layer; the upsampling layer upsamples the output of the first activation function layer; the second transposed convolution layer performs a transposed convolution operation on the output of the upsampling layer; the second activation function layer performs nonlinear processing on the output of the second transposed convolution layer; the decoder convolution layer performs a convolution operation on the output of the second activation function layer to obtain reconstructed image data.

6. The method according to claim 5, characterized in that The pseudo labeler comprises an input layer, a fully connected layer, an activation function layer, and an output layer connected end to end in sequence; The input layer receives the final features of the image data output by the Encoder unit and transmits them to the fully connected layer; The fully connected layer performs spatial mapping on the final features of the image data to generate the number of feature categories; An activation function layer generates a distribution probability of each feature category based on the number of feature categories through a Softmax function; The output layer generates anomaly labels based on the distribution probability of each feature category and transmits them to the anomaly detection module.

7. The method according to claim 6, characterized in that The anomaly detection module comprises: A reconstruction deviation calculation unit, used for calculating the reconstruction deviation based on the image reconstruction data and the image data of the product to be inspected; An abnormal deviation calculation unit, used for calculating the abnormal deviation based on the abnormal reconstruction data and the image data of the product to be detected; A pseudo-label deviation calculation unit, used for calculating the pseudo-label deviation based on the abnormal label and the real label of the pre-labeled image data of the product to be detected; A regularization deviation calculation unit, used for calculating a regularization deviation based on potential abnormal information; The abnormality positioning unit is used to fuse the reconstruction deviation, the abnormal deviation, the pseudo-label deviation and the regularization deviation to obtain a comprehensive deviation feature; retrieve the historical abnormal image with the highest similarity to the comprehensive deviation feature in the memory module, and determine the position of the abnormal area in the image of the product to be inspected based on the abnormal area of ​​the historical abnormal image; The anomaly classification unit is used to obtain the anomaly category based on the anomaly label output by the pseudo labeler.

8. The method according to claim 7, characterized in that The reconstruction deviation is calculated based on the image reconstruction data and the image data of the product to be inspected as shown in formula (1); Among them, n is the number of image data of the product to be detected, y i is the image data of the i-th product to be inspected, Reconstruct data for the i-th image. The calculation of the abnormal deviation based on the abnormal reconstruction data and the image data of the product to be inspected is shown in calculation formula (2); Among them, n is the number of image data of the product to be detected, a i is the image data of the i-th product to be inspected, Reconstruct the data for the i-th anomaly.

9. The method according to claim 1, characterized in that: The pseudo-label deviation is calculated based on the abnormal label and the true label of the pre-labeled image data of the product to be inspected as shown in formula (3); Among them, n is the number of image data of the product to be detected, y ic is the encoding of the true label of the i-th image data, Encode the pseudo label of the i-th image data.

10. The method according to claim 7, characterized in that The regularization deviation is calculated based on the potential abnormal information as shown in formula (4); L latent =D KL (q(z|xi)||p(z)) (4) Among them, D KL is the KL divergence, q(z|xi) is the potential abnormal information, and p(z) is the prior distribution of the abnormal information.

Images