Fine-grained industrial image anomaly detection and positioning method and system based on two-way grid reconstruction

Abstract

The invention relates to the field of computer vision and image processing, and discloses a two-way grid reconstruction-based fine-grained industrial image anomaly detection and positioning method and system, and the method comprises the following steps: enabling an input image to pass through a pre-trained feature extractor to obtain original features, and mapping the original features to global and local coordinate spaces, sampling is carried out in the trained normal and abnormal grids according to global and local coordinates, sampling results are fused and then preliminary reconstruction features are obtained through a CNN module, the preliminary reconstruction features in the previous step and original features are input into a feature refining module, and a final reconstruction result is obtained. The abnormal score and the positioning map are calculated based on the final reconstruction result and the original features, through two-way grid reconstruction, full utilization of available normal samples and synthetic abnormal samples is achieved, normal feature boundaries are refined, the accuracy of abnormal detection and positioning is improved, and the method has excellent performance on fine-grained anomalies.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method and system for fine-grained industrial image anomaly detection and positioning based on dual-path grid reconstruction. Background Art

[0002] Image anomaly detection and localization aims to identify and accurately segment abnormal regions in images, and is widely used in industrial inspection, medical imaging, and video surveillance. However, this task faces great challenges due to the scarcity of abnormal samples and the diversity of abnormal patterns (such as minor scratches in industrial production to severe structural damage). Under these challenges, more and more research focuses on developing unsupervised methods.

[0003] In industrial image anomaly detection, typical examples of unsupervised methods include PaDiM, SPADE, and PatchCore, which rely on an external vector database to store features extracted from normal samples. During inference, anomalies are identified by calculating the Euclidean distance between the test sample and the nearest neighbor in the database. Although these methods are effective, they have poor generalization capabilities due to their discrete feature storage method, and require the storage of a large number of diverse normal features, resulting in high space complexity and resource-intensive search operations. In addition, existing industrial image anomaly detection methods often perform poorly when dealing with fine-grained defects, mainly because they lack knowledge of real anomaly data, resulting in the defined normal data boundaries being not compact enough. Although some generative models such as DFMGAN and AnomalyDiffusion can synthesize more realistic anomalies and help to refine the normal data boundaries, the training of these models requires a lot of computing resources, limiting their widespread use in practical applications.

[0004] Therefore, how to solve the accuracy and efficiency problems existing in the above methods and use the knowledge of normal samples and synthetic abnormal samples to improve the performance of fine-grained anomaly detection has become a topic that needs to be studied. In addition, how to effectively synthesize diverse abnormal data under limited computing resources is also a problem that needs to be solved urgently. For this reason, this application proposes a method and system for fine-grained industrial image anomaly detection and positioning based on dual-path grid reconstruction. Summary of the invention

[0005] The object of the present invention is to provide a method and system for fine-grained industrial image anomaly detection and positioning based on dual-path grid reconstruction to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for fine-grained industrial image anomaly detection and positioning based on dual-path grid reconstruction, comprising the following steps:

[0007] Step S1, pre-processing the input sample image to be processed to unify the size and standardize it, then aligning the intermediate layer output through the pre-trained feature extraction network to obtain the original features, and then further synthesizing the abnormal features on the original features through the FBP module through the abnormal grid training stage, wherein the sample image to be processed includes an industrial product image without label information;

[0008] Step S2, inputting the original features in step S1 into the coordinate mapping module, and obtaining the local and global coordinates through the convolution layer mapping trained by the coordinate mapping module;

[0009] Step S3, using the local and global coordinates obtained in step S2, sampling is performed from the normal and abnormal grids, and the sampling results are fused and input into the CNN module to obtain preliminary reconstruction features, the normal and abnormal grids contain corresponding local and global grids respectively, and the local and global coordinates are used to sample in the local and global grids to obtain preliminary reconstruction features that take into account both local details and global information of the image;

[0010] Step S4, inputting the original features in step S1 and the preliminary reconstructed features in step S3 into a feature refinement module to obtain final reconstructed features, wherein the feature refinement module combines mean square error and cosine similarity to evaluate the pixel-level similarity between the original features and the preliminary reconstructed features, and guides the weighted fusion of the original features and the preliminary reconstructed features based on this similarity to obtain the final reconstructed features;

[0011] Step S5, using the difference between the final reconstructed features of step S4 and the original features of step S1 as an anomaly score to detect and locate anomalies;

[0012] Step S6: after receiving the inference request from the terminal device, the image to be tested is mapped to the feature space, and then reconstructed at the feature level to obtain reconstructed features, and the reconstructed features are compared with the original features to obtain the inference results.

[0013] Preferably, the feature extraction network pre-trained in step S1 includes deep convolutional neural networks such as ResNet, VGG, and Inception.

[0014] Preferably, the FBP anomaly synthesis module of step S1 synthesizes anomalies with controllable size, shape, intensity and position for normal samples at the feature level only during the abnormal grid training, and provides corresponding annotated mask images. Specifically, the FBP anomaly synthesis module operates through the following steps:

[0015] Define the block size as B, the block strength as I, and the block center coordinates (x c ,y c ), first, generate the initialization mask M:

[0016] M=zeros(2B+1,2B+1)

[0017] Where M is a (2B+1)×(2B+1) matrix with an initial value of 0. Then, generate a random walk mask: 1) select the initial position (x0, y0), x0=B, y0=B; 2) randomly select the number of walk steps N, whose value is randomly selected from B to 2B; 3) perform a random walk process, randomly select the value of Δx and Δy each time, and its value range is {-1, 0, 1}, and update the position (x k+1 ,y k+1 ), where x k+1 =x k +Δx,y k+1 =y k +Δy, and mark the corresponding position in the mask M as 1; finally, generate the abnormal block and paste it to the feature map: 1) Initialize the block paste tensor P, whose size is 1×1×H×W and the initial value is 0; 2) According to the mark of the mask M, paste the block with intensity I to the corresponding position of the feature map; 3) Perform Gaussian blur processing on the pasted tensor P to obtain P blurred ; 4) The Gaussian blurred tensor P blurred Paste onto the feature map.

[0018] Preferably, the coordinate mapping module in step S2 maps the original features to the corresponding coordinate space by learning the mapping relationship between the local and global coordinate spaces, wherein the local coordinate space is used to capture the local details of the image and the global coordinate space is used to capture the overall structure of the image.

[0019] Preferably, the normal and abnormal grid creation process of step S3 is as follows: 1) creating an uninitialized grid tensor; 2) initializing the grid tensor using Xavier normal distribution; 3) converting the initialized tensor into Parameter type and managing it using ParameterList.

[0020] As a preferred embodiment, the normal and abnormal grid training process of step S3 is as follows: 1) first freeze the normal grid parameters, use normal samples to synthesize abnormalities through the FBP module, and train the abnormal grid; 2) freeze the abnormal grid parameters and train the normal grid; 3) the coordinate mapping module in S2 and the CNN module in S3 are both trainable modules and participate in the training throughout the process; 4) the mean square error is used as the reconstruction loss for training the normal grid, and the formula is as follows:

[0021]

[0022] Where C represents the number of channels of the feature map, H and W represent the height and width of the feature map respectively, φ(x) represents the original feature in S1, Represents the final reconstructed features of S4; 5) The training abnormal grid also uses the above method to reconstruct the error, where φ(x) is replaced by the feature map after synthesizing the abnormality, and the truncated L1 loss is used as the contrast loss to refine the boundary between normal features and abnormal features. The formula is as follows:

[0023]

[0024] Where th represents the size of the buffer around the dividing line, D is a set of sample pair similarities constructed based on the mask, and d + The table represents the similarity of the positive sample pairs, d - Represents the similarity of the negative sample pair.

[0025] Preferably, the feature refinement module in step S4 evaluates the pixel-level similarity between the original feature and the preliminary reconstructed feature by calculating the mean square error (MSE) and cosine similarity, wherein MSE is used to capture the absolute difference at the pixel level, and cosine similarity is used to capture the directional similarity between feature vectors. Specifically, the feature refinement module is operated by the following steps:

[0026] First, calculate the original feature φ aligned (x) and preliminary reconstruction features The similarity between them is Sim, and the formula is as follows:

[0027]

[0028] where ∈ is an infinitely small constant,

[0029]

[0030] in, is an indicator function, and the value is 1 when the condition in the brackets is met. mse(·,·) and cosim(·,·) represent the calculation of mean square error and cosine similarity, respectively. The above similarity Sim is then applied to the reconstructed features to obtain the final reconstructed features.

[0031]

[0032] Among them, ⊙ represents element-by-element multiplication.

[0033] A system for a fine-grained industrial image anomaly detection and location method based on dual-path grid reconstruction according to any of the above items, the system comprising: a preprocessing module, used to unify the size and standardize the input image, select the intermediate layer output alignment as the original feature through a pre-trained feature extraction network, and during the abnormal grid training, use the FBP abnormal synthesis module to define the size, strength and position of the abnormal block at the normal sample feature level, use the random walk algorithm to generate a mask, and synthesize the abnormal block onto the feature map of the normal sample to obtain the original feature;

[0034] A processing module is used to obtain global coordinates and local coordinates of the original features through a coordinate mapping module; the global coordinates and local coordinates are input into a dual-path grid reconstruction module, the dual-path grid reconstruction module is composed of a normal grid and an abnormal grid, the normal grid and the abnormal grid respectively contain corresponding local and global grids, in each path, global and local feature sampling is obtained by grid sampling, wherein the global and local feature sampling are fused by splicing along the channel dimension, and then the features of the normal grid and the abnormal grid are weighted fused; the fused feature sampling is input into a CNN module to obtain preliminary reconstruction features; the preliminary reconstruction features and the original features are input into a feature refinement module, the pixel-level similarity between the original features and the preliminary reconstruction features is evaluated by calculating the mean square error (MSE) and cosine similarity, and the similarity result is used as a weight to fuse the original features and the preliminary reconstruction features to obtain the final reconstruction features; the anomaly score is calculated using the difference between the final reconstruction features and the original features to detect and locate the anomaly, wherein the anomaly score is the anomaly score of each pixel, and the image-level anomaly score is the maximum value of all pixel-level anomaly scores;

[0035] The query feedback module is used to receive the query image sent by the terminal device, map the query image to the feature space and convert it into a reconstructed feature generated by the model, determine the image-level and pixel-level anomaly scores based on the difference between the reconstructed feature and the original feature, and feedback the query result to the terminal device.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a fine-grained industrial image anomaly detection and positioning system based on dual-path grid reconstruction. The system first standardizes the input image and extracts original features through a pre-trained network. In the abnormal grid training stage, a feature map containing anomalies is generated through an FBP anomaly synthesis module. The processing module maps the original features to local and global coordinate spaces, and performs feature sampling and splicing through a dual-path grid reconstruction module to generate preliminary reconstructed features. The feature refinement module calculates mean square error (MSE) and cosine similarity, merges the preliminary reconstructed features with the original features to obtain final reconstructed features, calculates anomaly scores using the differences between the final reconstructed features and the original features, and determines pixel-level and image-level anomalies. The query feedback module receives a query image, maps it to a feature space, generates a reconstructed feature, and feeds back anomaly detection results based on the differences between the reconstructed features and the original features. The system can efficiently and accurately perform industrial defect detection and positioning, improve detection performance, and excel in inference speed and memory occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of one implementation of the industrial defect detection and positioning model of the present invention;

[0039] Figure 2 A schematic diagram of one implementation of the preprocessing module of the present invention;

[0040] Figure 3 This is a schematic diagram of the effect of the FBP abnormal synthesis module of the present invention;

[0041] Figure 4 It is a schematic diagram of the method flow of the present invention;

[0042] Figure 5 Schematic diagram of the system architecture of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Example

[0045] See also Figure 4 ,The method of fine-grained industrial image anomaly detection and localization based on dual-path grid reconstruction in the figure includes the following steps,

[0046] Step S1, pre-processing the input sample image to be processed to unify the size and standardize it, then aligning the intermediate layer output through the pre-trained feature extraction network to obtain the original features, and then further synthesizing the abnormal features on the original features through the FBP module through the abnormal grid training stage, wherein the sample image to be processed includes an industrial product image without label information;

[0047] Step S2, inputting the original features in step S1 into the coordinate mapping module, and obtaining the local and global coordinates through the convolution layer mapping trained by the coordinate mapping module;

[0048] Step S3, using the local and global coordinates obtained in step S2, sampling is performed from the normal and abnormal grids, and the sampling results are fused and input into the CNN module to obtain preliminary reconstruction features, the normal and abnormal grids contain corresponding local and global grids respectively, and the local and global coordinates are used to sample in the local and global grids to obtain preliminary reconstruction features that take into account both local details and global information of the image;

[0049] Step S4, inputting the original features in step S1 and the preliminary reconstructed features in step S3 into a feature refinement module to obtain final reconstructed features, wherein the feature refinement module combines mean square error and cosine similarity to evaluate the pixel-level similarity between the original features and the preliminary reconstructed features, and guides the weighted fusion of the original features and the preliminary reconstructed features based on this similarity to obtain the final reconstructed features;

[0050] Step S5, using the difference between the final reconstructed features of step S4 and the original features of step S1 as an anomaly score to detect and locate anomalies;

[0051] Step S6: after receiving the inference request from the terminal device, the image to be tested is mapped to the feature space, and then reconstructed at the feature level to obtain reconstructed features, and the reconstructed features are compared with the original features to obtain the inference results.

[0052] Furthermore, the abnormal grid trained with synthetic anomalies and the normal grid trained with normal samples are combined. Under the constraint of the abnormal grid, the normal feature boundary in the normal grid is further refined, thereby improving the detection performance of the model on fine-grained anomalies in complex scenes. Figure 3 As shown in the figure, the FBP anomaly synthesis module is used to synthesize diverse anomalies at the feature level to train the anomaly grid, obtain the knowledge of the abnormal samples, and thus guide the normal grid to further refine its normal feature boundaries to reconstruct higher quality normal features. The main method flow is as follows Figure 3 and Figure 5 As shown, including:

[0053] The sample image to be processed is input into the preprocessing module for preprocessing, including: unifying and standardizing the image size and then inputting it into the pre-trained feature extractor, aligning the intermediate layer features to the same size and then splicing them to obtain the original features. In the abnormal grid training stage, the abnormalities will be synthesized at the feature level as the abnormal original features through the FBP abnormal synthesis module based on the original features.

[0054] The preprocessing module mentioned in this embodiment can be implemented as a code program in actual application. First, the input RGB image is processed to a uniform size, and then the image after the uniform size is standardized to make it obey the standard normal distribution. Then, the middle layer features are taken through the pre-trained feature extractor, and the multiple middle layer features taken are aligned to the selected layer of features with the largest resolution and then spliced ​​together to obtain the original features. In addition, during the abnormal grid training, an abnormal block will be synthesized on the original feature. This includes: generating a mask area by random walk according to the position and size parameters, and then controlling the intensity of the synthesized abnormal block and the original feature through the intensity parameter to fuse them, and then refining the edge of the abnormal block through Gaussian blur to obtain the original feature with abnormality. For specific synthesis effects, please refer to Figure 3 .

[0055] The obtained original features are input into the coordinate mapping module to obtain global and local coordinates for subsequent grid sampling, wherein the coordinate mapping module includes a global coordinate mapping module and a local coordinate mapping module. The original features are mapped to the global coordinate space through the global coordinate mapping module and to the local coordinate space through the local coordinate mapping module. In the global coordinate mapping module, the input original features are first mapped to the new feature space through a fully connected layer (Linear), and then the Tanh activation function is applied to introduce nonlinear characteristics. Next, the features are further adjusted through the fully connected layer again, and finally the Tanh activation function is applied again. In the local coordinate mapping module, the original features are first linearly transformed between channels through a 1×1 convolution layer, and then the Tanh activation function is applied to introduce nonlinear characteristics. Next, the features are further adjusted through a 1×1 convolution layer again, and finally the Tanh activation function is applied again. Through the combination of global and local coordinate mapping modules, the system can effectively capture the local details and overall structure of the image, thereby achieving more accurate feature mapping.

[0056] The obtained global and local coordinates are input into the two-way grid reconstruction module to obtain preliminary reconstruction features, where the grid is trained as a coordinate function with infinite resolution, and the features corresponding to the output coordinates are output. The output features with infinite resolution are aggregated by nearby features in the grid according to the distance between the coordinates of the input features and the adjacent features. For example, when taking a one-dimensional grid sampling Channel

[0057] The output features of C are composed of a one-dimensional grid The adjacent value interpolation of is expressed as:

[0058] φ(v;G)=|vn|G[m]+|vm|G[n]

[0059]

[0060] in, is any input coordinate normalized to the grid resolution R, G[i] represents the feature of index i from the grid G, m and n are the indices to be referenced, where Represents the lower limit and upper limit operations respectively, by interpolating the 2 of the D-dimensional grid D values ​​(for example, a 2D grid interpolates the neighboring 2 coordinates based on the input coordinates 2 That is, 4 values), the above equation can be simply extended to a higher-dimensional grid, where the dual-path grid reconstruction module includes a normal grid and an abnormal grid, each grid is further subdivided into a global grid and a local grid, the global coordinates and the local coordinates are respectively input into the dual-path grid reconstruction module, and the grid sampling function is used to perform bilinear interpolation on the input global and local coordinates to generate a sampled feature map. For the grid sampling of normal features, the global coordinates and the local coordinates are input into the grid sampling function to obtain the global and local normal feature sampling results respectively; for the grid sampling of abnormal features, the global coordinates and the local coordinates are also input into the grid sampling function to obtain the global and local abnormal feature sampling results respectively. The sampling results are then normalized to eliminate the dimensional differences of the eigenvalues ​​and ensure the stability and consistency of subsequent processing. The normalized sampling results are input into the predefined convolution block for processing. The convolution block consists of multiple convolution layers, batch normalization layers, and activation functions. The sampled normal features are input into the normal convolution block for processing to obtain preliminary normal reconstruction features; the sampled abnormal features are input into the abnormal convolution block for processing to obtain preliminary abnormal reconstruction features. Finally, the preliminary reconstruction features are processed through a 1×1 convolution layer to further adjust the number of channels and dimensions of the features, and the abnormal reconstruction features and normal reconstruction features are fused to obtain preliminary reconstruction features.

[0061] The preliminary reconstructed features are input into the feature refinement module, and the original features and the preliminary reconstructed features are fused by calculating the similarity of the original features as weights to obtain the final reconstructed features. The feature refinement module evaluates the pixel-level similarity between the original features and the preliminary reconstructed features by calculating the mean square error MSE and cosine similarity. MSE is used to capture the absolute difference at the pixel level, and cosine similarity is used to capture the structural similarity between feature vectors. Specifically, the feature refinement module operates through the following steps:

[0062] First, calculate the original feature φaligned (x) and preliminary reconstruction features The similarity between them is Sim, and the formula is as follows:

[0063]

[0064] Where C is the number of channels of the feature map, ∈ is a small constant used to avoid division by zero,

[0065]

[0066] in, is an indicator function, which is assigned a value of 1 when the conditions in the brackets are met. mse(·,·) and cosim(·,·) represent the calculation of the mean square error and cosine similarity of each pixel, respectively. The above similarity Sim is then applied to the reconstructed features to obtain the final reconstructed features.

[0067]

[0068] Among them, ⊙ represents element-by-element multiplication.

[0069] The final reconstructed features are compared with the original features to obtain a pixel-level anomaly localization map, and then an image-level anomaly classification result is obtained based on the pixel-level anomaly localization map.

[0070] Among them, the difference between the final reconstructed feature and the original feature is used as the anomaly score to detect and locate the anomaly, and the difference between the two features is used to calculate the anomaly score by using the Euclidean distance. The specific implementation steps are as follows: first, the difference between the reconstructed image feature and the input original image feature is calculated, and the difference is defined as the reconstructed image feature minus the input original image feature; then, the difference is squared to obtain a square difference matrix; next, the square difference matrix is ​​summed in the feature dimension, that is, the channel dimension, and the square root is taken to obtain the Euclidean distance matrix of each pixel position, which reflects the degree of difference between the reconstructed image and the input original image in the feature space. In order to ensure that the spatial resolution of the difference matrix is ​​consistent with the input image, the Euclidean distance matrix is ​​further upsampled. The upsampling operation can Bilinear interpolation, nearest neighbor interpolation or other appropriate interpolation methods are used to ensure that the up-sampling result has the same spatial resolution as the input image. Finally, the Euclidean distance matrix is ​​the anomaly score matrix. By performing threshold processing on the anomaly score matrix, the abnormal area in the input image can be effectively detected and located. The threshold can be pre-set according to the specific application scenario or determined by experiment. In order to obtain the image-level positioning result, in this embodiment, the maximum value in the pixel-level anomaly score matrix is ​​compared with a preset threshold value. The specific steps are as follows: the maximum value in the pixel-level anomaly score matrix is ​​calculated, and the maximum value is compared with a preset threshold value. If the maximum value is greater than or equal to the threshold value, it is determined that there is an abnormality in the image; otherwise, it is determined that there is no abnormality in the image.

[0071] The working mode of this embodiment is similar to that of a commonly used target detection engine. When a user inputs an appearance image of an industrial product into the system, the system converts the image into original features through a preprocessing module, and then inputs these original features into a processing module to obtain the final reconstructed features. The system calculates the pixel-level anomaly score by comparing the difference between the original features of the product image and the final reconstructed features, and calculates the image-level anomaly classification result based on the maximum value of the pixel-level anomaly score of this sample, and finally returns the corresponding query result to the user. For example, when a user inputs a picture of a screw, the system returns the result: whether the screw in the image has a production defect or anomaly, and if so, the specific location of the defect will be given. That is, when a user inputs a picture, the system will inform the user whether the item in the picture has an industrial defect and the specific location of the defect. The model for detecting anomalies simultaneously considers the ability of normal grids that rely on normal samples and abnormal grids that rely on synthetic anomalies to discover abnormal patterns. The normal feature boundaries of the normal grid are refined at the feature level by the abnormal grid, which significantly enhances the reconstruction performance of the normal grid. The present invention adopts dual-path grid reconstruction and feature refinement technology to obtain better feature reconstruction results, thereby significantly improving the distinction between normal and abnormal.

[0072] In this embodiment, step S1 includes: processing the input RGB image to a uniform size, then standardizing the image after the uniform size so that it obeys the standard normal distribution, and then extracting the intermediate layer features through a pre-trained feature extractor, aligning the extracted multiple intermediate layer features to the selected layer of features with the largest resolution, and then splicing them together to obtain the original features.

[0073] In addition, during the training of the abnormal grid, the FBP abnormal synthesis technology is additionally used on the original features to synthesize abnormal blocks with controllable size, shape, intensity, and position at the feature level, and provide corresponding annotated mask images. Specifically, the FBP module operates through the following steps:

[0074] Define the block size as B, the block strength as I, and the block center coordinates (x c ,y c ), first, generate the initialization mask M:

[0075] M=zeros(2B+1,2B+1)

[0076] Where M is a (2B+1)×(2B+1) matrix with an initial value of 0. Then, a random walk mask is generated:

[0077] 1) Select the initial position (x0, y0), x0 = B, y0 = B;

[0078] 2) Randomly select the number of steps N, whose value is randomly selected from B to 2B;

[0079] 3) Perform a random walk process, randomly select the values ​​of Δx and Δy each time, and the value range is {-1, 0, 1}, and update the position (x k+1 ,y k+1 ), where x k+1 =x k +Δx,y k+1 =y k +Δy, and mark the corresponding position in mask M as 1.

[0080] Finally, the abnormal blocks are generated and pasted onto the feature map: 1) Initialize the block pasting tensor P, whose size is 1×1×H×W and the initial value is 0; 2) Paste the blocks with intensity I to the corresponding positions of the feature map according to the mark of the mask M; 3) Gaussian blur the pasted tensor P to obtain P blurred ; 4) The Gaussian blurred tensor P blurred Paste onto the feature map.

[0081] In this embodiment, step S2 includes: mapping the original features of the pre-trained feature extraction module to the global coordinate space through the global coordinate mapping module, and mapping them to the local coordinate space through the local coordinate mapping module; in the global coordinate mapping module, firstly mapping the input original features to a new feature space through a fully connected layer (Linear), then introducing nonlinear characteristics by applying the Tanh activation function, then further adjusting the features by the fully connected layer again, and finally applying the Tanh activation function again to ensure the nonlinearity of the output features; in the local coordinate mapping module, firstly performing a linear transformation between channels on the input features by a 1×1 convolution layer, then applying the Tanh activation function to introduce nonlinear characteristics, then further adjusting the features by the 1×1 convolution layer again, and finally applying the Tanh activation function again to ensure the nonlinearity of the output features. Through the combination of the above-mentioned global and local coordinate mapping modules, the system can effectively capture the local details and overall structure of the image, thereby achieving more accurate feature mapping.

[0082] In this embodiment, step S3 includes: inputting the global coordinates and the local coordinates into the dual-path grid reconstruction module respectively: using the grid_sample function provided by torch to perform bilinear interpolation on the feature map according to the input global and local coordinates to obtain the sampled feature map; for the grid sampling of normal features, inputting the global coordinates and the local coordinates into the grid_sample function to obtain the global and local normal feature sampling results respectively; for the grid sampling of abnormal features, also inputting the global coordinates and the local coordinates into the grid_sample function to obtain the global and local abnormal feature sampling results respectively; for the sampling The results are normalized to eliminate the dimensional differences of the feature values. Then, the normalized sampling results are input into the convolution block for processing: the predefined convolution block processes the sampled features, and the convolution block consists of multiple convolution layers, batch normalization layers, and activation functions (such as ReLU); the sampled normal features are input into the normal convolution block for processing to obtain preliminary normal reconstruction features; the sampled abnormal features are input into the abnormal convolution block for processing to obtain preliminary abnormal reconstruction features. Finally, after further adjusting the number of channels and dimensions of the features through a 1×1 convolution layer, the abnormal reconstruction features are used to constrain the normal reconstruction features to obtain preliminary reconstruction features.

[0083] In this embodiment, in step S4, the obtained preliminary reconstruction features are passed through a feature refinement module to obtain final reconstruction features;

[0084] Among them, the feature refinement module evaluates the pixel-level similarity between the original features and the preliminary reconstructed features by calculating MSE and cosine similarity. MSE is used to capture the absolute difference at the pixel level, and cosine similarity is used to capture the structural similarity between feature vectors. Specifically, the feature refinement module operates through the following steps:

[0085] First, calculate the original feature φ aligned (x) and preliminary reconstruction features The similarity between them is Sim, and the formula is as follows:

[0086]

[0087] Where C is the number of channels of the feature map, ∈ is a small constant used to avoid division by zero,

[0088]

[0089] in, is an indicator function, which is assigned a value of 1 when the condition in the brackets is met. mse(·, ·) and cosim(·, ·) represent the calculation of pixel-by-pixel mean square error and cosine similarity, respectively.

[0090] The above similarity Sim is then applied to the reconstructed features to obtain the final reconstructed features

[0091]

[0092] Among them, ⊙ represents element-by-element multiplication.

[0093] In this embodiment, in step S5, the final reconstructed features are compared with the original features to obtain a pixel-level anomaly localization map, and then an image-level anomaly classification result is obtained according to the pixel-level anomaly localization map, wherein what is compared is the feature space difference between the two images, and the feature space difference is achieved by calculating the anomaly score using the Euclidean distance, and the following steps are performed:

[0094] First, calculate the reconstructed image features And the input original image feature φ aligned The difference between (x) is defined as:

[0095]

[0096] The differences are then squared to obtain a squared difference matrix:

[0097] square_diff = diff 2

[0098] Next, the square difference matrix is ​​summed in the feature dimension (i.e., channel dimension) and the square root is taken to obtain the Euclidean distance matrix for each pixel position, which is defined as:

[0099]

[0100] The Euclidean distance matrix pred reflects the difference between the reconstructed image and the input original image in the feature space. In order to ensure that the spatial resolution of the difference matrix is ​​consistent with the input image, the Euclidean distance matrix is ​​further upsampled:

[0101] pred pixel = upsample(pred)

[0102] The upsampling operation may use bilinear interpolation, nearest neighbor interpolation or other appropriate interpolation methods to ensure that the upsampled matrix has the same spatial resolution as the input image. Finally, the Euclidean distance matrix pred pixel That is, the anomaly score matrix. By performing threshold processing on the anomaly score matrix, the abnormal area in the input image can be effectively detected and located. The threshold can be pre-set according to the specific application scenario or determined through experiments.

[0103] In order to obtain the image-level positioning result, in this embodiment, the pixel-level abnormal score matrix pred pixel This is achieved by comparing the maximum value in with the preset threshold. The specific steps are as follows:

[0104] img_score=max(pred pixel )

[0105] The image-level anomaly score img_score is compared with a preset threshold threshold. If img_score is greater than or equal to the threshold, it is determined that an anomaly exists in the image; otherwise, it is determined that no anomaly exists in the image.

[0106]

[0107] Among them, pred img It is a binary variable. When its value is 1, it indicates that there is an abnormality in the image. When its value is 0, it indicates that there is no abnormality in the image.

[0108] The advantages of the present invention are: by using spatial domain information and frequency domain information to jointly guide the training of the model, the dual-domain information complements each other and jointly discovers abnormal patterns in the image. The image preprocessing module highlights the high-frequency information that is easily lost in image reconstruction, and greatly avoids the model from seeing abnormalities during reasoning. The dual-domain feature selection module considers both the spatial domain and the frequency domain when modeling the normal image distribution. The dual-domain reconstruction not only captures the local details and global structure of the image, but also effectively utilizes the frequency domain information, thereby achieving higher quality image reconstruction, which directly determines the detection performance of the model.

[0109] It should be noted that this embodiment is not a simple calculation method, but can be applied to industrial production and assist in improving the production line of industrial products. For example, in practical applications, the method of this embodiment can be applied to a Figure 1 The system shown comprises:

[0110] The preprocessing module is used to unify the size of the input images and perform standardization. Through the pre-trained feature extraction network, the intermediate layer output is selected and aligned as the original feature. During the abnormal grid training, the FBP abnormal synthesis module is used to define the size, intensity and position of the abnormal block at the feature level of the normal sample, and the mask is generated using the random walk algorithm. The abnormal block is synthesized onto the feature map of the normal sample to obtain the original feature containing the abnormality.

[0111] The processing module maps the above original features to the local and global coordinate spaces through the coordinate mapping module. The coordinate mapping module includes two parts, local and global, which respectively perform local and global coordinate mapping of the features. Then, these coordinates are input into the dual-path grid reconstruction module. The global and local feature samplings are obtained by grid sampling. The global and local feature samplings are fused by splicing. The normal and abnormal grids contain the corresponding local and global grids respectively. Then, the spliced ​​feature samplings are input into a CNN module to obtain preliminary reconstructed features. Subsequently, the preliminary reconstructed features and the original features are input into the feature refinement module. The pixel-level similarity between the original features and the preliminary reconstructed features is evaluated by calculating the mean square error (MSE) and cosine similarity. The similarity results are used as weights to fuse the original features and the preliminary reconstructed features to obtain the final reconstructed features.

[0112] The query feedback module is used to receive the query image sent by the terminal device, map the query image to the feature space and convert it into the final reconstructed feature generated by the model, use the difference between the final reconstructed feature and the original feature as the anomaly score to detect and locate the anomaly, the anomaly score is the anomaly score of each pixel, the image-level anomaly score is determined by selecting the maximum value of all pixel-level anomaly scores, and feedback the query result to the terminal device.

[0113] Specifically, this embodiment is suitable for defect detection and positioning of industrial images, that is, the product image is converted into a reconstructed image through the pre-training module and the trained model. The model will further compare the gradient and color differences between the input product image and the output reconstructed image to detect and locate the anomaly, and then return the corresponding detection and positioning results. For example: the working method is similar to the currently commonly used image recognition. When the assembly line produces industrial parts, the machine on the assembly line takes a picture of the part, obtains the image and inputs it into the model. Then the model reconstructs the image and uses the gradient and color anomaly evaluation functions to evaluate whether the image is abnormal. Finally, the result is returned. This result can be used to determine whether the assembly line continues production, because when defective industrial products are produced, it means that there is a problem with the assembly line. It may be necessary to stop production and repair the machine, otherwise it will cause a waste of resources. It can also be used to guide the subsequent repair of faulty machines. The location where the defect occurs may also contain information about the fault.

[0114] This embodiment is suitable for defect detection and positioning of industrial images. Specifically, the product image obtains original features through the preprocessing module, and is reconstructed into final reconstructed features through the processing module. The model further compares the difference between the original features and the final reconstructed features to detect and locate anomalies, and returns corresponding results. For example, on the production line, the machine takes pictures of industrial parts, obtains images and inputs them into the model. The model then reconstructs the image at the feature level and uses an MSE-based anomaly evaluation function to determine whether there is a problem with the image. Finally, the system returns the detection result to decide whether to continue production. If the product has defects, this indicates that the production line may have a fault and may need to be stopped for maintenance to prevent waste of resources. At the same time, the defect information can also provide guidance for subsequent maintenance of the machine. The defect location and its morphology can reveal the fault information.

[0115] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0116] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for fine-grained industrial image anomaly detection and localization based on dual-path grid reconstruction, characterized in that: The steps include: Step S1, pre-processing the input sample image to be processed to unify the size and standardize it, then aligning the intermediate layer output through the pre-trained feature extraction network to obtain the original features, and then further synthesizing the abnormal features on the original features through the FBP module through the abnormal grid training stage, wherein the sample image to be processed includes an industrial product image without label information; Step S2, inputting the original features in step S1 into the coordinate mapping module, and obtaining the local and global coordinates through the convolution layer mapping trained by the coordinate mapping module; Step S3, using the local and global coordinates obtained in step S2, sampling is performed from the normal and abnormal grids, and the sampling results are fused and input into the CNN module to obtain preliminary reconstruction features, the normal and abnormal grids contain corresponding local and global grids respectively, and the local and global coordinates are used to sample in the local and global grids to obtain preliminary reconstruction features that take into account both local details and global information of the image; Step S4, inputting the original features in step S1 and the preliminary reconstructed features in step S3 into a feature refinement module to obtain final reconstructed features, wherein the feature refinement module combines mean square error and cosine similarity to evaluate the pixel-level similarity between the original features and the preliminary reconstructed features, and guides the weighted fusion of the original features and the preliminary reconstructed features based on this similarity to obtain the final reconstructed features; Step S5, using the difference between the final reconstructed features of step S4 and the original features of step S1 as an anomaly score to detect and locate anomalies; Step S6: after receiving the inference request from the terminal device, the image to be tested is mapped to the feature space, and then reconstructed at the feature level to obtain reconstructed features, and the reconstructed features are compared with the original features to obtain the inference results.

2. According to claim 1, a method for fine-grained industrial image anomaly detection and positioning based on dual-path grid reconstruction is characterized in that: The feature extraction network pre-trained in step S1 includes deep convolutional neural networks such as ResNet, VGG, and Inception.

3. The method for fine-grained industrial image anomaly detection and location based on dual-path grid reconstruction according to claim 2 is characterized in that: The FBP anomaly synthesis module of step S1 synthesizes anomalies with controllable size, shape, intensity and position for normal samples at the feature level only during the abnormal grid training, and provides corresponding annotated mask images. Specifically, the FBP anomaly synthesis module operates through the following steps: Define the block size as B, the block strength as I, and the block center coordinates (x c ,y c ), first, generate the initialization mask M: M=zeros(2B+1,2B+1) Where M is a (2B+1)×(2B+1) matrix with an initial value of 0. Then, generate a random walk mask: 1) select the initial position (x0, y0), x0=B, y0=B; 2) randomly select the number of walk steps N, whose value is randomly selected from B to 2B; 3) perform a random walk process, randomly select the value of Δx and Δy each time, and its value range is {-1, 0, 1}, and update the position (x k+1 ,y k+1 ), where x k+1 =x k +Δx,y k+1 =y k +Δy, and mark the corresponding position in the mask M as 1; finally, generate the abnormal block and paste it to the feature map: 1) Initialize the block paste tensor P, whose size is 1×1×H×W and the initial value is 0; 2) According to the mark of the mask M, paste the block with intensity I to the corresponding position of the feature map; 3) Perform Gaussian blur processing on the pasted tensor P to obtain P blurred ; 4) The Gaussian blurred tensor P blurred Paste onto the feature map.

4. The method for fine-grained industrial image anomaly detection and location based on dual-path grid reconstruction according to claim 3 is characterized by: The coordinate mapping module in step S2 maps the original features to the corresponding coordinate space by learning the mapping relationship between the local and global coordinate spaces, wherein the local coordinate space is used to capture the local details of the image, and the global coordinate space is used to capture the overall structure of the image.

5. The method for fine-grained industrial image anomaly detection and location based on dual-path grid reconstruction according to claim 4 is characterized in that: The normal and abnormal grid creation process of step S3 is as follows: 1) creating an uninitialized grid tensor; 2) initializing the grid tensor using Xavier normal distribution; 3) converting the initialized tensor into Parameter type and managing it using ParameterList.

6. The method for fine-grained industrial image anomaly detection and location based on dual-path grid reconstruction according to claim 5 is characterized by: The normal and abnormal grid training process of step S3 is as follows: 1) first freeze the normal grid parameters, use normal samples to synthesize abnormalities through the FBP module, and train the abnormal grid; 2) freeze the abnormal grid parameters and train the normal grid; 3) the coordinate mapping module in S2 and the CNN module in S3 are both trainable modules and participate in the training throughout the process; 4) the mean square error is used as the reconstruction loss for training the normal grid, and the formula is as follows: Where C represents the number of channels of the feature map, H and W represent the height and width of the feature map respectively, φ(x) represents the original feature in S1, represents the final reconstructed features of S4; 5) The training abnormal grid also uses the above method to reconstruct the error, where φ(x) is replaced by the feature map after synthesizing the abnormality, and the truncated L1 loss is used as the contrast loss to refine the boundary between normal features and abnormal features. The formula is as follows: Where th represents the size of the buffer around the dividing line, D is a set of sample pair similarities constructed based on the mask, and d + The table represents the similarity of the positive sample pairs, d - Represents the similarity of the negative sample pair.

7. The method for fine-grained industrial image anomaly detection and location based on dual-path grid reconstruction according to claim 6 is characterized in that: The feature refinement module in step S4 evaluates the pixel-level similarity between the original feature and the preliminary reconstructed feature by calculating the mean square error (MSE) and cosine similarity, wherein MSE is used to capture the absolute difference at the pixel level, and cosine similarity is used to capture the directional similarity between feature vectors. Specifically, the feature refinement module operates through the following steps: First, calculate the original feature φ aligned (x) and preliminary reconstruction features The similarity between them is Sim, and the formula is as follows: where ∈ is an infinitely small constant, in, is an indicator function, and the value is 1 when the conditions in the brackets are met. mse(·,·) and cosim(·,·) represent the calculation of mean square error and cosine similarity, respectively. The above similarity Sim is then applied to the reconstructed features to obtain the final reconstructed features. Among them, ⊙ represents element-by-element multiplication.

8. A system for fine-grained industrial image anomaly detection and positioning method based on dual-path grid reconstruction according to any one of claims 1 to 7, characterized in that: The system includes: a preprocessing module, which is used to unify the size of the input image and standardize it, select the intermediate layer output after alignment as the original feature through the pre-trained feature extraction network, and use the FBP abnormal synthesis module during the abnormal grid training to define the size, strength and position of the abnormal block at the normal sample feature level, use the random walk algorithm to generate a mask, and synthesize the abnormal block onto the feature map of the normal sample to obtain the original feature; A processing module is used to obtain global coordinates and local coordinates of the original features through a coordinate mapping module; the global coordinates and local coordinates are input into a dual-path grid reconstruction module, the dual-path grid reconstruction module is composed of a normal grid and an abnormal grid, the normal grid and the abnormal grid respectively contain corresponding local and global grids, in each path, global and local feature sampling is obtained by grid sampling, wherein the global and local feature sampling are fused by splicing along the channel dimension, and then the features of the normal grid and the abnormal grid are weighted fused; the fused feature sampling is input into a CNN module to obtain preliminary reconstruction features; the preliminary reconstruction features and the original features are input into a feature refinement module, the pixel-level similarity between the original features and the preliminary reconstruction features is evaluated by calculating the mean square error (MSE) and cosine similarity, and the similarity result is used as a weight to fuse the original features and the preliminary reconstruction features to obtain the final reconstruction features; the anomaly score is calculated using the difference between the final reconstruction features and the original features to detect and locate the anomaly, wherein the anomaly score is the anomaly score of each pixel, and the image-level anomaly score is the maximum value of all pixel-level anomaly scores; The query feedback module is used to receive the query image sent by the terminal device, map the query image to the feature space and convert it into a reconstructed feature generated by the model, determine the image-level and pixel-level anomaly scores based on the difference between the reconstructed feature and the original feature, and feed back the query result to the terminal device.

Images