Abnormal independent feature component extraction method, device and storage medium based on convolutional neural network

Abstract

The present invention discloses a method, device and storage medium for extracting abnormal independent feature components based on a convolutional neural network. The method can separate the foreground abnormal features from the background features of a picture. By introducing a reconstruction network, a generative adversarial network and a domain adversarial network, the independence of the foreground abnormal features and the background features is ultimately ensured, and the purpose of extracting abnormal independent feature components is achieved. The method effectively solves the technical problems existing in the adaptability and flexibility of the abnormal detection method. In addition, the present invention also provides an abnormal independent feature component extraction device and a storage medium based on the above method.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormality detection using image recognition, and in particular to a method, device and storage medium for extracting abnormal independent feature components based on a convolutional neural network. Background Art

[0002] Image anomaly detection is one of the core tasks in the field of computer vision, and its goal is to identify anomalies or outliers from a dataset. Anomaly detection plays an important role in many fields. Currently, most research focuses on unsupervised anomaly detection, and the main methods can be divided into two categories: feature embedding and reconstruction-based methods.

[0003] Among the feature embedding methods, the memory-based method is the most effective in image anomaly detection tasks. This method requires additional storage space to store image features, and only requires sampling of the collected normal image features for inference. During the inference process, the features of the test image are compared with the features in the memory, and the degree of abnormality of the test image is determined by calculating the spatial distance between the test image features and the features in the memory (such as Euclidean distance or cosine similarity).

[0004] The reconstruction-based method mainly inputs the normal image or abnormal image into the reconstruction network and guides the network training through the reconstruction loss function. Finally, the reconstruction network can restore the image in a way similar to the original normal image. In the inference stage, the model compares the original image with the reconstructed image to generate a prediction result.

[0005] For the above methods, the memory-based feature embedding method relies on a memory containing normal image features. The memory is generated from the training data and reflects the normal patterns in the training set. If normal patterns that have not been seen in the training set appear in the test image, these patterns may not find matching features in the memory, resulting in false positives. The reconstruction-based method reconstructs the input image by training the network. However, the ability of the reconstruction network depends on the diversity and representativeness of the training data. If the training data is not enough to cover all normal patterns, the reconstruction network may not be able to effectively reconstruct normal images that have not been seen in the training set, resulting in false positives. Therefore, both methods have limitations in adaptability and flexibility. For the same foreground abnormal feature, once its background domain changes, the performance of the above method models will drop sharply. In order to adapt to the new background domain distribution, these models need to be retrained, which is not only time-consuming and labor-intensive, but also increases the consumption of computing resources. Summary of the invention

[0006] In view of this, one of the objects of the present invention is to provide a method for extracting abnormal independent feature components based on convolutional neural networks, which can effectively solve the technical problems of the above-mentioned existing image anomaly detection methods being limited in adaptability and flexibility.

[0007] One of the purposes of the present invention is achieved through the following technical solutions:

[0008] The abnormal independent feature component extraction method based on convolutional neural network includes the following steps:

[0009] Step S1: Collect and classify image data, divide the data into two data sets: training set and test set according to task requirements, each data set contains normal images and abnormal images, and then preprocess the image data;

[0010] Step S2: Separate the foreground abnormal features and background features of the image data using a feature separation network based on a convolutional neural network, the network includes two parts: feature extraction and feature separation;

[0011] Step S3: Based on the fact that there are no abnormal defects in the normal image, the foreground abnormal features of the normal image separated in step S2 are Apply constraints; the constraint method is to minimize the mean of the absolute values ​​of the foreground abnormal features of the normal image so that the foreground abnormal features of the normal image Approaching zero, the foreground feature map approaches the all-zero feature map;

[0012] Step S4: A domain adversarial network is used to suppress information confusion between foreground abnormal features and background features. The foreground abnormal features and background features separated in step 2 are respectively connected to different domain adversarial networks. For foreground features, it is encouraged to retain semantic information related to foreground defects and isolate irrelevant background information into background features. The processing strategy for background features is opposite to that for foreground features.

[0013] Step S5: Reconstruction loss is introduced to further constrain the feature extraction and separation network, and a reconstruction network is constructed to connect the separated foreground abnormal features and background features for reconstruction. L1 loss is used as the reconstruction loss, and the integrity and complementarity of the foreground abnormal features and background features are ensured by minimizing the loss.

[0014] Step S6: In order to enhance the expressive power of foreground abnormal features and background features, a discriminator is introduced to simulate a generative adversarial network based on the reconstruction network in step S5. A new abnormal image is generated by adding the foreground abnormal features of the abnormal image to the background features of the normal image. A new normal image is generated by adding the background features of the abnormal image to the all-zero feature map. The original image and the generated image are input into the discriminator D for authenticity identification, and the adversarial loss is used to enhance the expressive power of the foreground abnormal features and background features.

[0015] Furthermore, in step S1, the classified and labeled image data is to assign the abnormal image a foreground abnormal category label and a background domain category label, the normal image is only assigned a background domain category label, and the foreground abnormal category label is set to 0; preprocessing refers to uniformly formatting the data input form into [image, background label, foreground abnormal label].

[0016] Furthermore, in step S2, the feature extraction part uses a pre-trained ResNet50 model, and removes the last two linear layers as the backbone network for image feature extraction;

[0017] The feature separation part is composed of a convolution layer with a 1x1 convolution kernel and a Gumbel Softmax function. The convolution layer with a 1x1 convolution kernel is used to change the number of channels and increase the complexity of feature representation. The Gumbel Softmax function is used to generate a foreground mask, and the background mask is obtained by subtracting the foreground mask from 1.

[0018] After the image enters the feature extraction part, the original features are obtained. After the original features pass through the convolution layer of the feature separation part, the separated features containing richer feature information are obtained.

[0019] Finally, the foreground anomaly feature F is obtained by multiplying the separation feature with the foreground mask, and the background feature B is obtained by multiplying the separation feature with the background mask.

[0020] Furthermore, in step S3, the foreground feature map of the image is a three-dimensional tensor with a size of , then the normal image foreground abnormal features It is expressed as follows:

[0021] ;

[0022] in, Indicates the number of channels, each channel is a The elements in each matrix are , Indicates Channels at position ( , ), take the absolute value of the foreground feature value at each position and sum it up, we can get , whose expression is:

[0023] ;

[0024] By summing up the foreground feature values ​​of all channels and then averaging them, we can get the mean of the absolute value of the foreground feature of the entire image. , whose expression is:

[0025] ;

[0026] During the training process, each batch of N samples contains M normal image samples, so it is necessary to calculate the mean of the M normal image samples to obtain the loss function , whose expression is:

[0027] ;

[0028] in, Indicates The normal image sample is Channels at position ( , ) is the foreground feature value. During the training process, the batch loss function is minimized by minimizing the absolute mean. , the parameters of the feature separation network will be adjusted so that the foreground abnormal features of the normal image The absolute value mean of gradually decreases, thereby compressing the foreground abnormal feature value of the normal image to approach zero.

[0029] Furthermore, in step S4, the domain adversarial module is provided with a foreground anomaly classifier and background domain classifier Two classification heads are used to predict the abnormal category and background domain category of the original image through feature input;

[0030] First, the foreground anomaly features separated in step S2 are input into the foreground anomaly classifier ,pass The probability distribution of the abnormal category to which the original image belongs is obtained, and its expression is:

[0031] ;

[0032] Among them, F i Indicates The foreground features of samples, , Represents the foreground feature F i The probability distribution of the corresponding abnormal category;

[0033] Foreground anomaly class classification loss of foreground anomaly features The cross entropy loss function is used to measure the predicted probability distribution of abnormal categories. The abnormal category label of the original image obtained in step S1 The difference between them is expressed as:

[0034] ;

[0035] Then the foreground abnormal feature is connected to the background domain classifier At the same time, a gradient reversal layer GRL is added in front of the background domain classifier. GRL acts as an identity function in the process of forward propagation and does not change the input value. It is expressed as: = , but the sign of the gradient is reversed during back propagation, expressed as:

[0036] ;

[0037] in is the identity matrix;

[0038] Background Domain Classifier To distinguish different background domain categories, The foreground feature input The probability distribution of the background domain category to which the original image belongs is obtained, and its expression is:

[0039] ;

[0040] in, Representing foreground features The probability distribution of the corresponding background domain category;

[0041] Classification loss of background domain categories based on foreground anomaly features The cross entropy loss function is also used to measure the predicted probability distribution The background label of the original image obtained in step S1 The difference between them is expressed as:

[0042] ;

[0043] By minimizing and This ensures that the foreground anomaly features do not contain any information related to the background domain.

[0044] Furthermore, in step S5, the reconstruction network uses a deconvolution layer to upsample the input feature map; each deconvolution layer is followed by a batch normalization layer and a ReLU activation function to improve the stability of the reconstruction; the input image The foreground abnormal feature F and the background feature B are obtained through step S2. The reconstruction module recombines the separated foreground abnormal feature F and the background feature B, and generates a reconstructed image through a reconstruction network based on a convolutional neural network CNN. , the reconstruction loss function uses the standard L1 reconstruction loss, image The reconstruction loss is expressed as , whose expression is:

[0045] ;

[0046] Reconstruction module by minimizing The loss causes the reconstruction network to force the foreground abnormal features and background features to better reconstruct the original image, thereby prompting the feature separation network to learn more detailed and accurate features when separating foreground abnormalities and background.

[0047] Furthermore, in step S6, the discriminator body uses convolutional layers for downsampling, each convolutional layer is followed by batch normalization and ReLU activation function, and finally the fully connected layer outputs the true and false classification results;

[0048] The background feature B of the normal image G Add the foreground abnormal feature F of the abnormal image B Generate new abnormal images through reconstruction network ,Will and provide F B Abnormal image input discriminator To make a distinction;

[0049] The background feature B of the abnormal image B Add all zero features of the same dimension and generate a new normal image through the reconstruction network ,Will Identify the normal image input with the same background category as the abnormal image To make a distinction;

[0050] The goal of the discriminator is to maximize the ability to distinguish between real images and generated images, using cross entropy loss as the discriminator loss L D , whose expression is:

[0051] ;

[0052] in represents the output of the discriminator for real images, represents the output of the discriminator for the generated images.

[0053] Furthermore, during the training process, the generated images Improve fidelity, making Maximize the closeness to the real image to deceive the discriminator, and reconstruct the network as the generator using L2 loss as the adversarial loss , reducing the discriminator's sensitivity to real images X and generated images The output difference between the two is expressed as:

[0054] ;

[0055] During the training process, the reconstruction network and the discriminator are trained adversarially. The reconstruction network attempts to generate images that are as close to the real image as possible to deceive the discriminator, while the discriminator maximizes the ability to distinguish between real images and generated images. The training goal of the reconstruction network is to minimize the probability that the generated image is recognized as fake by the discriminator, that is, to minimize , and the training goal of the discriminator is to maximize the ability to distinguish between real images and generated images, that is, to maximize L D .

[0056] The second object of the present invention is to provide an abnormal independent feature component extraction device based on a convolutional neural network, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method described above when executing the computer program.

[0057] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

[0058] The beneficial effects of the present invention are:

[0059] (1) By introducing the ideas of reconstruction network, generative adversarial network and domain adversarial network, the feature separation network is constrained by reconstruction loss, generative adversarial loss and domain adversarial loss to ensure the independence and mutual exclusivity of foreground abnormal features and background features after feature separation;

[0060] (2) The independence of foreground anomaly features and background features enables the model to maintain stable performance in different background domains, improves the generalization ability of the model, reduces the need for retraining, and reduces computing resources and time costs.

[0061] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description and the preceding claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0063] Figure 1 It is a method flow chart of the abnormal independent feature component extraction method based on convolutional neural network of the present invention;

[0064] Figure 2 This is a schematic diagram of a specific process of separating foreground abnormal features and background features in step S2 described in the embodiment;

[0065] Figure 3 This is a schematic diagram of a specific process of using a domain adversarial network in step S4 of the embodiment to suppress information confusion between foreground abnormal features and background features;

[0066] Figure 4 A schematic diagram of using a reconstruction network in step S5 of the embodiment;

[0067] Figure 5 This is a schematic diagram of simulating a generative adversarial network based on the reconstructed network in step S6 of the embodiment. DETAILED DESCRIPTION

[0068] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0069] like Figure 1 As shown, the abnormal independent feature component extraction method based on convolutional neural network of the present invention comprises the following steps:

[0070] Step S1: Collect and classify image data, and divide the data into two data sets: training set and test set according to task requirements. Each data set contains normal images and abnormal images to ensure that the model can be exposed to various types of data during the training and testing stages. Then, the image data is preprocessed.

[0071] In this embodiment, the classification and labeling of image data is to assign a foreground abnormality category label and a background domain category label to abnormal images, only assign a background domain category label to normal images, and set the foreground abnormality category label to 0; preprocessing refers to uniformly formatting the data input form into [image, background label, foreground abnormality label] for model training;

[0072] Step S2: Separate the foreground abnormal features and background features of the image data using a feature separation network based on a convolutional neural network. The network includes two parts: feature extraction and feature separation. In this embodiment, the feature extraction part uses a pre-trained ResNet50 model, and removes the last two linear layers as a backbone network for image feature extraction.

[0073] The feature separation part is composed of a convolution layer with a 1x1 convolution kernel and a Gumbel Softmax function. The convolution layer with a 1x1 convolution kernel is used to change the number of channels and increase the complexity of feature representation. The Gumbel Softmax function is used to generate a foreground mask, and the background mask is obtained by subtracting the foreground mask from 1.

[0074] After the image enters the feature extraction part, the original features are obtained. After the original features pass through the convolution layer of the feature separation part, the separated features containing richer feature information are obtained.

[0075] Finally, the foreground anomaly feature F is obtained by multiplying the separation feature with the foreground mask, and the background feature B is obtained by multiplying the separation feature with the background mask.

[0076] Step S3: Based on the fact that there are no abnormal defects in the normal image, the foreground abnormal features of the normal image separated in step S2 are Apply constraints; the constraint method is to minimize the mean of the absolute values ​​of the foreground abnormal features of the normal image so that the foreground abnormal features of the normal image Approaching zero, the foreground feature map approaches the all-zero feature map;

[0077] Specifically, the foreground feature map of an image is a three-dimensional tensor with size , then the normal image foreground abnormal features It is expressed as follows:

[0078] ;

[0079] in, Indicates the number of channels, each channel is a The elements in each matrix are , Indicates Channels at position ( , ), take the absolute value of the foreground feature value at each position and sum it up, we can get , whose expression is:

[0080] ;

[0081] By summing up the foreground feature values ​​of all channels and then averaging them, we can get the mean of the absolute value of the foreground feature of the entire image. , whose expression is:

[0082] ;

[0083] During the training process, each batch of N samples contains M normal image samples, so it is necessary to calculate the mean of the M normal image samples to obtain the loss function , whose expression is:

[0084] ;

[0085] in, Indicates The foreground feature value of the normal image sample at the position (j, k) in the cth channel. During the training process, the batch loss function is minimized by minimizing the absolute mean , the parameters of the feature separation network will be adjusted so that the foreground abnormal features of the normal image The absolute value mean of gradually decreases, thus compressing the foreground abnormal feature value of the normal image to approach zero;

[0086] Step S4: In order to suppress the information confusion between the foreground abnormal features and the background features, the idea of ​​the domain adversarial network is adopted. In this embodiment, the foreground abnormal features and background features separated in step S2 are respectively connected to different domain adversarial networks. For the foreground features, it is encouraged to retain the semantic information related to the foreground defects, and the irrelevant background information is isolated into the background features. The processing strategy for the background features is opposite to that for the foreground features.

[0087] In this embodiment, the domain adversarial network is constructed by the following technical solutions. The domain adversarial module is provided with a foreground anomaly classifier and background domain classifier Two classification heads are used to predict the abnormal category and background domain category of the original image through feature input;

[0088] First, the foreground anomaly features separated in step S2 are input into the foreground anomaly classifier ,pass The probability distribution of the abnormal category to which the original image belongs is obtained, and its expression is:

[0089] ;

[0090] in, Indicates The foreground features of samples, , Represents the foreground feature F i The probability distribution of the corresponding abnormal category;

[0091] Foreground anomaly class classification loss of foreground anomaly features The cross entropy loss function is used to measure the predicted probability distribution of abnormal categories. The abnormal category label of the original image obtained in step 1 The difference between them is expressed as:

[0092] ;

[0093] Then the foreground abnormal feature is connected to the background domain classifier At the same time, a gradient reversal layer GRL is added in front of the background domain classifier. GRL acts as an identity function in the process of forward propagation and does not change the input value. It is expressed as: = , but the sign of the gradient is reversed during back propagation, expressed as:

[0094] ;

[0095] in is the identity matrix;

[0096] Background Domain Classifier Used to distinguish different background domain categories, the foreground features after GRL are input The probability distribution of the background domain category to which the original image belongs is obtained, and its expression is:

[0097] ;

[0098] in, Represents the foreground feature F i The probability distribution of the corresponding background domain category;

[0099] Classification loss of background domain categories based on foreground anomaly features The cross entropy loss function is also used to measure the predicted probability distribution The background label of the original image obtained in step S1 The difference between them is expressed as:

[0100] ;

[0101] minimize The gradient reversal layer reverses the gradient during back propagation, making it more difficult for the background domain classifier to try to predict the category of the background domain, which forces the separation of purer foreground anomaly features in step S2, so that the foreground anomaly features do not contain any relevant information about the background domain.

[0102] For the background features separated in step S2, similar to the method of constraining the foreground abnormal features, the background features are input into the foreground abnormal classifier and the background domain classifier, but the gradient reversal layer is placed before the foreground abnormal classifier, making it more difficult for the background features to predict the abnormal category of the foreground, and making the separated background features purer and not containing relevant information about the foreground abnormality.

[0103] During the training process, by continuously minimizing the foreground anomaly classification loss and the background domain classification loss, the foreground anomaly features are encouraged to retain the semantic information related to the foreground anomaly, and the irrelevant background information is isolated into the background features, so that the foreground anomaly feature F can eventually determine the foreground anomaly category, but cannot determine the category of the background domain; for the background features, it is encouraged to retain the semantic information related to the background domain, and isolate the irrelevant foreground anomaly information into the foreground anomaly features, so that the background feature B can eventually determine the category of the background domain, but cannot determine the category of the foreground anomaly.

[0104] Step S5: Introduce reconstruction loss to further constrain the feature extraction and separation network, build a reconstruction network, connect the separated foreground abnormal features and background features for reconstruction, use L1 loss as the reconstruction loss, and ensure the integrity and complementarity of the foreground abnormal features and background features by minimizing the loss. The reconstruction process requires that the foreground features and background features contain complete image information together, which requires the separation network to ensure the integrity and complementarity of the foreground features and background features when separating features, thereby avoiding information loss.

[0105] In this embodiment, the reconstruction network uses a deconvolution layer to upsample the input feature map; each deconvolution layer is followed by a batch normalization layer and a ReLU activation function to improve the stability of the reconstruction; the input image The foreground abnormal feature F and the background feature B are obtained through step S2. The reconstruction module recombines the separated foreground abnormal feature F and the background feature B, and generates a reconstructed image through a reconstruction network based on a convolutional neural network CNN. , the reconstruction loss function uses the standard L1 reconstruction loss, image The reconstruction loss is expressed as , expressed as:

[0106] ;

[0107] Reconstruction module by minimizing The reconstruction loss forces the foreground anomaly features and background features to better reconstruct the original image. As a supervisory signal, the reconstruction loss guides the network to continuously adjust the feature separation method during the training process, prompting the feature separation network to learn more detailed and accurate features when separating foreground anomalies and background.

[0108] Step S6: In order to enhance the expressive power of foreground abnormal features and background features, the idea of ​​generative adversarial network is adopted. On the basis of the reconstruction network in step S5, a discriminator is introduced to simulate the generative adversarial network. A new abnormal image is generated by adding the foreground abnormal features of the abnormal image to the background features of the normal image. A new normal image is generated by adding the background features of the abnormal image to the all-zero feature map. The original image and the generated image are input into the discriminator D for authenticity identification. The expressive power of foreground abnormal features and background features is enhanced by using adversarial loss.

[0109] The discriminator body uses convolutional layers for downsampling, each convolutional layer is followed by batch normalization and ReLU activation function, and finally a fully connected layer outputs the true and false classification results;

[0110] The background feature B of the normal image G Add the foreground abnormal feature F of the abnormal image B Generate new abnormal images through reconstruction network ,Will and provide F B Abnormal image input discriminator To make a distinction;

[0111] The background feature B of the abnormal image B Add all zero features of the same dimension and generate a new normal image through the reconstruction network ,Will Identify the normal image input with the same background category as the abnormal image To make a distinction;

[0112] The goal of the discriminator is to maximize the ability to distinguish between real images and generated images, using cross entropy loss as the discriminator loss L D , whose expression is:

[0113] ;

[0114] in represents the output of the discriminator for real images, represents the output of the discriminator for the generated image;

[0115] During the training process, the generated images Improve fidelity, making Maximize the closeness to the real image to deceive the discriminator, and reconstruct the network as the generator using L2 loss as the adversarial loss , reducing the discriminator's sensitivity to real images X and generated images The output difference between the two is expressed as:

[0116] ;

[0117] During the training process, the reconstruction network and the discriminator are trained adversarially. The reconstruction network attempts to generate images that are as close to the real image as possible to deceive the discriminator, while the discriminator tries to improve its ability to distinguish between real images and generated images. The training goal of the reconstruction network is to minimize the probability that the generated image is recognized as fake by the discriminator, that is, to minimize , and the training goal of the discriminator is to maximize the ability to distinguish between real images and generated images, that is, to maximize L D Through this adversarial training mechanism, not only can The fidelity of the foreground image can also be improved, and the expression ability of the foreground abnormal features and the background features can be enhanced, so that the foreground abnormal features and the background features can more accurately reflect the information of the original image.

[0118] Through the comprehensive application of the above steps, the independence of foreground anomaly features and background features is ultimately ensured, achieving the purpose of extracting abnormal independent feature components, thereby greatly improving the generalization ability of the model. Regardless of how the image background changes, this method can maintain stable detection of foreground anomaly features, reducing the need for retraining and saving a lot of time and computing resources. This method is suitable for a variety of application scenarios, especially in areas that require efficient and reliable image anomaly detection.

[0119] The main purpose of the present invention is to overcome the limitation of the above technology in the generalization ability of the background domain, and propose a method for extracting abnormal independent feature components based on convolutional neural network. The method can separate the foreground abnormal features from the background features of the picture. If the input picture is a normal picture, the foreground abnormal features will be constrained to be a full-0 feature map. By introducing the ideas of domain adversarial network, reconstruction network and generative adversarial network, the foreground abnormal features and background features are independent and mutually exclusive. The purpose of the domain adversarial module is to suppress the information confusion between the foreground abnormal features and the background features. The module is provided with two classification heads, a foreground abnormal classifier and a background domain classifier, which are used to predict the abnormal category and the background domain category of the original image respectively. The foreground abnormal features are connected to the foreground abnormal classifier, and the abnormal related information in the foreground abnormal features is strengthened by predicting the abnormal category of the image; at the same time, the background domain classifier is connected, and the background related information in the foreground abnormal features is reduced by the gradient reversal layer, so as to prevent the foreground abnormal features from containing background information, and the processing strategy for the background features is opposite to that of the foreground features. The purpose of the reconstruction module is to ensure the integrity and complementarity of the foreground abnormal features and the background features by reconstructing the input image, thereby avoiding information loss. Based on the reconstruction module, the discriminator is introduced to construct a generative adversarial network. The reconstruction network attempts to generate images as realistic as possible to deceive the discriminator, while the discriminator tries to improve the ability to distinguish between real images and generated images. Through the adversarial training of the two, the expression ability of foreground abnormal features and background features is enhanced, and the authenticity of the generated images is improved. Through the comprehensive application of the above technologies, the independence of foreground abnormal features and background features is finally ensured, and the purpose of extracting abnormal independent feature components is achieved. In this way, in practical applications, such as bottle production for industrial anomaly detection, for specific foreground anomaly categories (such as scratches, cracks, etc.), even if these anomalies appear on different background materials (such as glass, plastic, etc.), there is no need to retrain the model. The model can directly separate surface abnormal defects and pass this information to the production line control system to eliminate unqualified products. This method effectively solves the technical problems existing in the adaptability and flexibility of anomaly detection methods.

[0120] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0121] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0122] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0123] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An abnormal independent feature component extraction method based on convolutional neural network, characterized by: The method comprises the following steps: Step S1: Collect and classify image data, divide the data into two data sets: training set and test set according to task requirements, each data set contains normal images and abnormal images, and then preprocess the image data; Step S2: Separate the foreground abnormal features and background features of the image data using a feature separation network based on a convolutional neural network, the network includes two parts: feature extraction and feature separation; Step S3: Based on the fact that there are no abnormal defects in the normal image, the foreground abnormal feature F of the normal image separated in step S2 is G Apply constraints; the constraint method is to minimize the mean of the absolute values ​​of the foreground abnormal features of the normal image so that the foreground abnormal features F of the normal image G Approximate to zero, so that the foreground feature map approaches the all-zero feature map; Step S4: A domain adversarial network is used to suppress the information confusion between foreground abnormal features and background features. The foreground abnormal features and background features separated in step S2 are respectively connected to different domain adversarial networks. For foreground features, it is encouraged to retain semantic information related to foreground defects and isolate irrelevant background information into background features. The processing strategy for background features is opposite to that foreground features. In step S3, the foreground feature map of the image is a three-dimensional tensor with a size of , then the normal image foreground abnormal features It is expressed as follows: ； in, Indicates the number of channels, each channel is a The elements in each matrix are , Indicates Channels at position ( , ), take the absolute value of the foreground feature value at each position and sum it up to get , whose expression is: ； Sum and average the foreground feature values ​​of all channels to get the mean of the absolute value of the foreground feature of the entire image. , whose expression is: ； During the training process, each batch of N samples contains M normal image samples, so it is necessary to calculate the mean of the M normal image samples to obtain the loss function , whose expression is: ； in, Indicates that the i-th normal image sample is in Channels at position ( , ) is the foreground feature value. During the training process, the batch loss function L is minimized by minimizing the absolute mean F , the parameters of the feature separation network will be adjusted so that the foreground abnormal feature F of the normal image G The absolute value mean of gradually decreases, thus compressing the foreground abnormal feature value of the normal image to approximately zero; Step S5: Introduce reconstruction loss to further constrain the feature extraction and separation network, build a reconstruction network, connect the separated foreground abnormal features and background features for reconstruction, use L1 loss as the reconstruction loss, and ensure the integrity and complementarity of the foreground abnormal features and background features by minimizing the loss; in step S5, the reconstruction network uses a deconvolution layer to upsample the input feature map; each deconvolution layer is followed by a batch normalization layer and a ReLU activation function to improve the stability of the reconstruction; input image The foreground abnormal feature F and the background feature B are obtained through step S2. The reconstruction module recombines the separated foreground abnormal feature F and the background feature B, and generates a reconstructed image through a reconstruction network based on a convolutional neural network CNN. , the reconstruction loss function uses the standard L1 reconstruction loss, image The reconstruction loss is expressed as , whose expression is: ； Reconstruction module by minimizing The loss causes the reconstruction network to force the foreground anomaly features and background features to better reconstruct the original image, thereby prompting the feature separation network to learn more detailed and accurate features when separating foreground anomalies and background; Step S6: In order to enhance the expressive power of foreground abnormal features and background features, a discriminator is introduced to simulate a generative adversarial network based on the reconstruction network in step S5, and a new abnormal image is generated by adding the foreground abnormal features of the abnormal image to the background features of the normal image, and a new normal image is generated by adding the background features of the abnormal image to the all-zero feature map. The original image and the generated image are input into the discriminator D to identify the true and false, and the expressive power of the foreground abnormal features and background features is enhanced by using the adversarial loss; in the step S6, the discriminator body uses a convolutional layer to downsample, each convolutional layer is followed by batch normalization and ReLU activation function, and finally the fully connected layer outputs the true and false classification results; The background feature B of the normal image G Add the foreground abnormal feature F of the abnormal image B Generate new abnormal images through reconstruction network ,Will and provide F B Abnormal image input discriminator To make a distinction; The background feature B of the abnormal image B Add all zero features of the same dimension and generate a new normal image through the reconstruction network ,Will Identify the normal image input with the same background category as the abnormal image To make a distinction; The goal of the discriminator is to maximize the ability to distinguish between real images and generated images, using cross entropy loss as the discriminator loss , whose expression is: ； in represents the output of the discriminator for real images, represents the output of the discriminator for the generated images.

2. The method for extracting abnormal independent feature components based on a convolutional neural network according to claim 1, characterized in that: In the step S1, the classification and labeling of image data is to assign foreground abnormality category labels and background domain category labels to abnormal images, only assign background domain category labels to normal images, and set the foreground abnormality category label to 0; preprocessing refers to uniformly formatting data input in the form of [image, background label, foreground abnormality label].

3. The method for extracting abnormal independent feature components based on a convolutional neural network according to claim 1, characterized in that: In step S2, the feature extraction part uses a pre-trained ResNet50 model, and removes the last two linear layers as the backbone network for image feature extraction; The feature separation part is composed of a convolution layer with a 1x1 convolution kernel and a Gumbel Softmax function. The convolution layer with a 1x1 convolution kernel is used to change the number of channels and increase the complexity of feature representation. The Gumbel Softmax function is used to generate the foreground mask, and the background mask is obtained by subtracting the foreground mask from 1; After the image enters the feature extraction part, the original features are obtained. After the original features pass through the convolution layer of the feature separation part, the separated features containing richer feature information are obtained. Finally, the foreground anomaly feature F is obtained by multiplying the separation feature with the foreground mask, and the background feature B is obtained by multiplying the separation feature with the background mask.

4. The method for extracting abnormal independent feature components based on a convolutional neural network according to claim 1, characterized in that: In step S4, the domain adversarial module is provided with a foreground anomaly classifier and background domain classifier Two classification heads are used to predict the abnormal category and background domain category of the original image through feature input; First, the foreground anomaly features separated in step S2 are input into the foreground anomaly classifier ,pass The probability distribution of the abnormal category to which the original image belongs is obtained, and its expression is: ； in, Indicates The foreground features of samples, , Represents foreground features The probability distribution of the corresponding abnormal category; Foreground anomaly class classification loss of foreground anomaly features The cross entropy loss function is used to measure the predicted probability distribution of abnormal categories. The abnormal category label of the original image obtained in step S1 The difference between them is expressed as: ； Then the foreground abnormal feature is connected to the background domain classifier At the same time, a gradient reversal layer GRL is added in front of the background domain classifier. GRL acts as an identity function and does not change the input value during the forward propagation process, which is expressed as: = , but the sign of the gradient is reversed during back propagation, expressed as: ； in is the identity matrix; Background Domain Classifier To distinguish different background domain categories, The foreground feature input The probability distribution of the background domain category to which the original image belongs is obtained, and its expression is: ； in, Represents foreground features The probability distribution of the corresponding background domain category; Classification loss of background domain categories based on foreground anomaly features The cross entropy loss function is also used to measure the predicted probability distribution The background label of the original image obtained in step S1 The difference between them is expressed as: ； By minimizing and This ensures that the foreground anomaly features do not contain any information related to the background domain.

5. The method for extracting abnormal independent feature components based on convolutional neural network according to claim 1, characterized in that: During the training process, the generated images Improve fidelity, making Close to the real image to fool the discriminator, the reconstruction network is used as the generator using L2 loss as the adversarial loss , reduce the discriminator's sensitivity to real images X and generated images The output difference between the two is expressed as: ； During the training process, the reconstruction network and the discriminator are trained adversarially. The reconstruction network attempts to generate images that are as close to the real image as possible to deceive the discriminator, while the discriminator maximizes the ability to distinguish between real images and generated images. The training goal of the reconstruction network is to minimize the probability that the generated image is recognized as fake by the discriminator, that is, to minimize , and the training goal of the discriminator is to maximize the ability to distinguish between real images and generated images, that is, to maximize .

6. An abnormal independent feature component extraction device based on a convolutional neural network, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images