Unsupervised anomaly detection method based on pixel single-point structure and multi-element pairing logic

Abstract

The application belongs to the technical field of industrial image defect detection, and discloses an unsupervised anomaly detection method based on pixel single-point and multi-element pairing. The method comprises the following steps: S1, constructing a structural feature extraction branch network and a logical feature extraction branch network; S2, inputting images for training into the structural feature extraction branch network and the logical feature extraction branch network respectively to form a structural feature memory bank and a logical feature memory bank; S3, obtaining test pixel features and test logical features corresponding to a test image, obtaining a structural anomaly score map and a logical anomaly score map of the test image; and S4, fusing the structural anomaly score map and the logical anomaly score map to obtain a total anomaly score map of the test image, and determining a defect position according to the anomaly score map. Through the application, the problems of surface structure and position logic anomaly in product images are solved.

Description

Technical Field

[0001] This invention belongs to the technical field of industrial image defect detection, and more specifically, relates to an unsupervised anomaly detection method based on single-point and multi-element pairing of pixels. Background Technology

[0002] In actual industrial manufacturing processes, various unforeseen circumstances, such as machine malfunctions, transportation damage, and worker errors, often lead to substandard industrial products. These substandard products, besides surface defects, may also exhibit logically anomalous defects such as incorrect product placement, missing packaging, or discrepancies between the ordered and delivered goods. To improve production and delivery efficiency, quality inspection methods have shifted from traditional manual inspection to vision-based automated inspection, with anomaly detection being a representative task. From production and packaging to product delivery, not only may structural anomalies occur in the product's surface quality, but logical anomalies may also arise, such as incorrect product placement or mismatched packaging and products. Examples include missing or incorrectly matched screws in bags, inconsistent filling of juice in glass bottles, or incorrect label placement. Because abnormal situations are complex and unpredictable, relying solely on product surface quality as a method for determining whether a product is qualified is insufficient. Currently, the more mature defect detection methods for structural anomalies are insufficient to handle scenarios involving the detection of logical anomalies in products. Therefore, the detection problem of samples with logical relationships is a more complex problem of joint structural and logical anomaly detection, which means that it is necessary not only to detect surface structural anomalies in the product image in the sample, but also to detect potential positional logical anomalies in the sample image.

[0003] Therefore, the joint detection model of structural and logical anomalies can effectively utilize the image structure and logical relationship information of image pixels of anomaly samples, further improving the performance of the anomaly detection model, which is crucial for industrial production quality inspection. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an unsupervised anomaly detection method based on single pixel point and multi-element pairing, which solves the problem of surface structure and positional logic anomalies in product images.

[0005] To achieve the above objectives, according to the present invention, an unsupervised anomaly detection method based on single-point and multi-element pairing of pixels is provided, the method comprising the following steps:

[0006] S1 Constructs a structural feature extraction branch network for extracting pixel features from an image and a logical feature extraction branch network for extracting logical features;

[0007] S2 inputs the images used for training into the structural feature extraction branch network and the logical feature extraction branch network respectively, thereby extracting the pixel features and logical features of each image. The pixel features of all images form a structural feature memory bank, and the logical features of all images form a logical feature memory bank.

[0008] S3. The image to be tested is input into the structural feature extraction branch network and the logical feature extraction branch network respectively to obtain the test pixel features and test logical features corresponding to the image to be tested. The maximum distance score between the test pixel features and the pixel features in the structural feature memory is calculated to obtain the anomaly score of the image to be tested, thereby obtaining the structural anomaly score map of the image to be tested. The global consistency between the test logical features and the logical features in the logical feature memory is calculated to obtain the anomaly score of the image to be tested, thereby obtaining the logical anomaly score map of the image to be tested.

[0009] S4 merges the structural anomaly score map with the logical anomaly score map to obtain the overall anomaly score map of the image under test. The anomaly score at each point in the anomaly score map is compared with a preset threshold. The position where the score is greater than the preset threshold is the location of the defect, and the other position is normal. This is used to determine the location of the defect in the image under test.

[0010] More preferably, in step S1, both the structural feature extraction branch network and the logical feature extraction branch network adopt the Wide ResNet50 network pre-trained on the ImageNet dataset.

[0011] More preferably, in step S2, the logical feature extraction branch network forms a logical feature memory according to the following steps:

[0012] S21 For the logical feature extraction branch network, extract one or more sets of pixel feature pairs from each feature layer;

[0013] S22 In each feature layer, the pixel feature pairs are connected according to a preset number of pixel pairs to form multi-dimensional pixel pairs, thereby obtaining multi-dimensional pixel pairs in all feature layers;

[0014] S23 Align the multi-element pixels of different feature layers, and all aligned multi-element pixels form a logical feature memory library.

[0015] More preferably, in step S3, the alignment employs a multi-scale feature fusion method.

[0016] More preferably, in step S3, the anomaly score in the structural anomaly score graph is calculated according to the following formula:

[0017]

[0018]

[0019] in, It is the structural anomaly score. It tests pixel features. Is with The structural features of training pixels with the highest similarity.

[0020] More preferably, in step S3, the anomaly score in the logic anomaly score graph is calculated according to the following formula:

[0021]

[0022]

[0023] in, It is a logical anomaly score. It tests the logic features. Is with The training pixel logical features with the highest similarity. More preferably, in step S4, the total anomaly score of the image to be tested is calculated according to the following formula:

[0024]

[0025] in, It is the structural anomaly score. It is a logical anomaly score.

[0026] More preferably, in step S2, after the structural feature memory and logical feature memory are formed, data processing is required on the elements in each memory to remove unqualified data.

[0027] More preferably, the data processing employs a greedy subsampling strategy.

[0028] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0029] 1. This method considers both individual image pixels and implicit logical information between image pixel pairs to address practical application issues in automated production line inspection of industrial products, such as whether the combination of workpiece quantity meets requirements and whether the product label type or location conforms to specifications, which require simultaneous consideration of structural and logical defects.

[0030] 2. The method provided by this invention enables unsupervised joint detection of structural and logical anomalies / defects, that is, during the training phase, only normal sample information can be used to achieve the joint detection of anomalies in local structure and global logic in the test sample during the testing phase. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the training phase constructed according to a preferred embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the construction of multi-pixel pairs according to a preferred embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the test phase constructed according to a preferred embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0035] To address the limitation of existing anomaly detection methods, which can only detect structural anomalies in the workpiece under test but cannot detect higher-level logical anomalies, an unsupervised anomaly detection method based on the logical relationship information between pixel pairs is proposed. The aim is to simultaneously utilize the information of individual pixels and the relationship information of paired pixels to model the potential logical or geometric relationships in the anomaly-free sample, thereby comparing and calculating the distance with the sample under test, and thus identifying structural and logical anomalies in the sample under test.

[0036] Furthermore, the model network structure of the unsupervised anomaly detection method based on pixel-to-pixel logical relationship information proposed in this invention mainly consists of a structural feature extraction branch network and a logical feature extraction branch network. The structural feature extraction branch network is a network model pre-trained on a large natural image dataset, and the features it extracts are highly discriminative. This invention further performs core set dimensionality reduction sampling on these features, which can eliminate redundant information and reduce inference time. There are certain logical relationships between pixels at different locations in an image, such as the matching relationship between the quantity of a product package and the number of cells (e.g., whether there are pins in each cell of the packaging box), and between the product label and the product itself (e.g., the juice label and the juice color). The logical feature extraction branch network extracts logical features from the logical relationship information between multiple pixel features in the image. The extracted features can consider long-distance binary and multi-dimensional logical relationships between image pixels. By modeling the logical relationship features, logical anomalies in the image can be identified (e.g., incorrect cable terminal location, mismatch between icon and juice type). Therefore, the structural feature extraction branch network and the logical feature extraction branch network in the model network structure proposed in this invention enable the model to jointly detect structural and logical anomalies in the test samples. In summary, this invention considers both structural and logical anomalies. It fits and learns the distribution of normal data features through two branches: the structural feature extraction network has a better ability to identify structural anomalies, while the logical feature extraction network has a better ability to identify logical anomalies.

[0037] When using the structural feature extraction branch network to extract image features, the pre-trained model can be used directly. However, before feature extraction in the logical feature extraction branch network, since pixels are independently distributed and there are no fixed relationships between different pixels, manual establishment of connections between pixel features is required. Therefore, this invention combines multi-scale pixel feature blocks of varying distances, feeding the concatenated pixel pairs into the network model for relational feature extraction to model the positional logical relationships in the image, and then storing the extracted logical features in a feature memory.

[0038] After extracting features using structural feature extraction branches and logical feature extraction branches respectively, they can be used to jointly detect structural anomalies and logical anomalies. To obtain the final result of joint detection, the anomaly score maps output by the two branch networks are added together and fused to obtain the final anomaly score map.

[0039] Specifically:

[0040] like Figure 1 As shown, the above method includes two feature extraction branch networks: a structural feature extraction branch network and a structural feature extraction branch network. Logical feature extraction branch network Among them, the structural feature extraction branch network Used for detecting structural anomalies, logical feature extraction branch network Used to detect logical anomalies.

[0041] This invention includes a training phase and a testing phase, wherein the training phase is as follows: Figure 1 ,include:

[0042] (1) Training phase for submodules with structural anomalies:

[0043] Images of normal industrial products Using structural features as input, branch networks are extracted. Extract the pixel block structure feature information of normal data, and then extract the normal pixel features. Aggregated into a memory bank Then, using a greedy subsampling method, we set... We use a subset of features to represent the majority of data features as much as possible. Each time, we select a point from this subset, find the furthest point within that subset, and sample and store it. This means finding the furthest point locally, i.e., the optimal solution, to eliminate redundant features. The final result is a structural feature memory used for detecting structural anomalies. .

[0044] (2) Training phase for sub-modules with logical anomalies:

[0045] Images of normal industrial products Using structural features as input, branch networks are extracted. Extracting pixel features from normal data at different feature layers as follows:

[0046] (1)

[0047] Furthermore, 1) we utilize logical features to extract branch networks. Extract one or more sets of pixel feature pairs at different locations from each feature layer, 2) and connect ( Forming multi-pixel pairs As shown in equation (2)

[0048] (2)

[0049] Multi-pixel pairs Composed of multiple sets of pixel pairs, Indicates the number of pixel pairs. For example... Figure 2 As shown, this example uses 4 sets of pixel pairs: Top and bottom Logical feature pixel pairs in the vicinity The specific definition is shown in equation (3).

[0050] (3)

[0051] And in the middle and both ends Logical feature pixel pairs in the vicinity The specific definition is shown in equation (4).

[0052] (4)

[0053] The vicinity can be left or right. , or up and down , Local structural pixel features that are adjacent in location, such as Represents the pixel features at the top center position. This represents the pixel features at the bottom right position.

[0054] The minimum angle formed by the two pixel feature blocks of a pixel pair with the horizontal or vertical direction is The included angle is specified in this invention. Pixel feature blocks can be randomly selected and paired within this included angle range. Multiple normal pixel blocks of different lengths and distances are merged and connected using the above method. To form a multi-pixel pair (the number of pixel blocks in a pixel pair can be greater than or equal to three; for ease of description of the method proposed in this invention and to simplify the annotation symbols), use... This represents the total number of pixel pairs (indicating the total number of pairings), used to model the relationships between pixels in an image. Specifically, pixels with a distance greater than half the distance of the farthest pixel pair are considered long-distance pixel pairs, while those less than that distance are considered short-distance pixel pairs. This is used to construct multi-scale logical features at different scales. ,

[0055] 3) Through multi-scale feature fusion The module fuses the logical features formed by different feature layers after aligning the sampled features from the upper and lower layers, as shown in equation (5).

[0056] (5)

[0057] Forming multi-feature pixel pairs Then, it is sent to the memory of normal image logical feature aggregation. Then, using the greedy subsampling method mentioned above in the structural anomaly detection section, redundant logical features are removed, ultimately resulting in a logical feature memory for detecting logical anomalies.

[0058] Structural feature memory and logical feature memory Together they form a feature memory bank The definition is as follows:

[0059] (6)

[0060] Testing phase, such as Figure 3 The details are as follows:

[0061] Test image Input structure feature extraction branch network Logical feature extraction branch network The two networks respectively obtain their corresponding structural features. and logical features The structural feature extraction branch network has already stored the pixel-level structural features of normal images into the structural feature memory during the training phase. During testing, the pixel features of the test image are calculated. and Normal pixel features The maximum distance score between pixels is used to estimate the anomaly score of the test image, and the structural anomaly score map of the test image is obtained by calculating the anomaly score of each pixel. The logical feature extraction branch network has already associated normal image pixel pairs during the training phase and stored their logical features in the logical feature memory. During testing, features are calculated by pairing image pixels. and The global consistency among normal pixel pairs is used to estimate the anomaly score of the test image, and the logical anomaly score map of the test image is obtained by calculating the consistency of each pixel pair. Finally, the structural anomaly score map... Logical anomaly score chart The final abnormal score is obtained by fusion. Anomaly score map as a test image.

[0062] Furthermore, during the training phase of the structural feature extraction branch network, the process of acquiring structural features is as follows: Figure 3 The partial branch network above illustrates this. First, normal samples are decomposed into a pixel-level feature set. , here use

[0063] (7)

[0064] To represent normal training samples In the pre-trained network In the j-th layer Dimensional position and The feature at that location is called a feature block. That is, a pixel feature block.

[0065] For all normal samples, the training samples Structural feature memory Simply defined as

[0066] (8)

[0067] To ensure inference speed and improve detection efficiency, we use a core set subset sampling mechanism to reduce the feature memory. Conceptually, the goal of core set selection is to find a subset. , making Solutions to the above problems and The solution is close to the one mentioned above.

[0068] (9)

[0069] This is the feature library we obtained after sampling from the core set. For ease of illustration, the feature library below will still use the same name. and This indicates that the actual features are all sampled from the core set.

[0070] By calculating the test feature block and Each nearest neighbor The maximum distance score between them is used to estimate the test image. Structural anomaly score map

[0071] (10)

[0072] (11)

[0073] For each test feature block, To find feature blocks in the structural feature library.

[0074] During the logic anomaly training phase, the process of acquiring logic features is as follows: Figure 3 The logical feature extraction branch network is shown below.

[0075] Normal samples are decomposed into pixel-level feature sets The pre-trained network extracts normal pixel features from product images at different locations, such as... , Pairing up , Pairing up , Pairing up , Pairing, at the angle between pixel features and horizontal or vertical angles Under the premise of random composition Groups of pixels are joined by merging. Use pixel-to-logic features This allows us to obtain the logical features of the dependency relationships between long and short distance pixel pairs at different scales. After multi-scale feature fusion ( The logical features following ) are represented as The definition is as follows:

[0076] (12)

[0077] Then store it in the logical feature memory bank. Used here This represents a pair of feature pixels in the same network layer. This represents the pre-trained network. Multiple different first layer, This represents a pair of logical feature pixels after multi-scale feature fusion.

[0078] For all normal samples, the training samples Logical feature memory Simply defined as

[0079] (13)

[0080] With a feature memory bank of logical description We calculate the features in the test feature block set. Logical features of paired pixels and Logical feature pixels of each nearest neighbor pixel The maximum distance score between the two is used to calculate the logistic anomaly score of the test image.

[0081] (14)

[0082] (15)

[0083] The final anomaly score map

[0084] (16)

[0085] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An unsupervised anomaly detection method based on pixel single-point and multi-element pairing, characterized in that, The method comprises the following steps: S1, constructing a structural feature extraction branch network for extracting pixel features of an image and a logical feature extraction branch network for logical feature extraction; S2, inputting images for training into the structural feature extraction branch network and the logical feature extraction branch network respectively to extract pixel features and logical features of each image, the pixel features of all images forming a structural feature memory bank, and the logical features of all images forming a logical feature memory bank; S3, inputting a to-be-tested image into the structural feature extraction branch network and the logical feature extraction branch network to obtain test pixel features and test logical features corresponding to the to-be-tested image, calculating a maximum distance score of a distance between the test pixel features and the pixel features in the structural feature memory bank to obtain an abnormal score of the to-be-tested image, thereby obtaining a structural abnormal score map of the to-be-tested image, and calculating a global consistency between the test logical features and the logical features in the logical feature memory bank to obtain an abnormal score of the to-be-tested image, thereby obtaining a logical abnormal score map of the to-be-tested image; S4, fusing the structural abnormal score map and the logical abnormal score map to obtain a total abnormal score map of the to-be-tested image, comparing an abnormal score at each position in the abnormal score map with a preset threshold, and determining a position of a defect in the to-be-tested image according to a result of the comparison. In step S2, the logical feature extraction branch network forms a logical feature memory bank according to the following steps: S21, extracting one or more groups of pixel feature pairs in each feature layer of the logical feature extraction branch network; S22, connecting the pixel feature pairs in each feature layer according to a preset number of pixel pairs to form multi-pixel pairs, thereby obtaining multi-pixel pairs in all feature layers; S23, aligning the multi-pixel pairs in different feature layers, and forming a logical feature memory bank from all aligned multi-pixel pairs.

2. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing according to claim 1, characterized in that, In step S1, the structural feature extraction branch network and the logical feature extraction branch network both use a Wide ResNet50 network pre-trained on an ImageNet dataset.

3. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing of claim 1, wherein, In step S3, the alignment uses a multi-scale feature fusion method.

4. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing according to claim 1 or 2, characterized in that, In step S3, the abnormal score in the structural abnormal score map is calculated according to the following relationship: wherein, is a structure anomaly score, is a test pixel structure feature, is a training pixel structure feature having the greatest similarity to the test pixel structure feature.

5. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing according to claim 1 or 2, characterized in that, In step S3, the abnormal score in the logical abnormal score map is calculated according to the following relationship: wherein, is the logical anomaly score, is the test pixel logical feature, is the training pixel logical feature with the greatest similarity to the test pixel logical feature.

6. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing according to claim 1 or 2, characterized in that, In step S4, the total abnormal score map of the to-be-tested image is calculated according to the following relationship: wherein, is a structural anomaly score, is a logical anomaly score.

7. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing according to claim 1 or 2, characterized in that, In step S2, after the structural feature memory bank and the logical feature memory bank are formed, data processing is required for elements in each memory bank to eliminate unqualified data.

8. The unsupervised anomaly detection method based on pixel single-point and multi-element pairing of claim 7, wherein, The data processing uses a sub-sampling method based on a greedy strategy.

Images