An anomaly detection and localization method and system based on adaptive residual algorithm

Abstract

This invention provides an anomaly detection and localization method and system based on an adaptive residual algorithm, relating to the fields of industrial image processing and computer vision. The method includes the following steps: acquiring an image to be detected; extracting category labels, multiple main image patch features, and auxiliary image patch features from the image using a dual encoder, and generating fusion query features; selecting multiple candidate main prototypes and multiple candidate auxiliary prototypes based on the category labels, and integrating the candidate main prototypes to obtain a candidate main prototype library; updating the candidate main prototype library to obtain a dynamic main prototype library; generating fusion reconstruction features based on the main image patch features and the dynamic main prototype library; generating a hybrid residual based on the fusion query features and the fusion reconstruction features; and inputting the hybrid residual into a lightweight segmentation head to generate a pixel-level anomaly map, enabling the output of anomaly determination and defect location. Using this method, the computational load of each step can be significantly reduced, balancing accuracy and inference efficiency, and adapting to edge devices and industrial quality inspection scenarios.

Description

Technical Field

[0001] This invention relates to the fields of industrial imaging and computer vision, specifically to an anomaly detection and localization method and system based on an adaptive residual algorithm. Background Technology

[0002] In industrial production, visual quality inspection typically requires the inspection system to determine whether a product has defects and to pinpoint the specific location of those defects in an image. Traditional supervised learning methods require acquiring a large number of defect images for different defect types and annotating them at the pixel level. However, defects in actual production are usually characterized by low frequency, randomness, and constant change, with significant morphological differences between different defects, making it difficult to exhaustively enumerate and collect sufficient samples in advance. In contrast, normal product images are easier to obtain. Therefore, using normal samples to establish a normal visual distribution and then identifying images or local areas that deviate from this distribution has high engineering value.

[0003] Among existing unsupervised anomaly detection methods, one type relies on matching with a normal feature library. This type of method uses a pre-trained encoder to extract dense image patch features from normal images and stores them in a repository. During detection, it calculates the distance between each image patch feature of the image to be detected and the normal features in the repository, using the nearest neighbor distance as the degree of anomaly. While this approach can retain a rich set of normal features, as the training samples, product categories, or image resolution increase, it requires storing and retrieving a large number of normal image patch features, resulting in significant storage, memory access, and nearest neighbor search overhead.

[0004] Another type of method is based on feature reconstruction. This type of method trains encoders, decoders, or Transformer structures to generate corresponding normal reconstructed features based on the image to be detected, and then judges anomalies based on the difference between the input features and the reconstructed features. It can generate corresponding normal references for the image to be detected, but it usually relies on a relatively complex attention network to implicitly complete prototype learning or feature reconstruction, resulting in a high number of model parameters and computational cost, and the formation process of dynamic prototypes is not explicit enough.

[0005] Furthermore, while using a single self-supervised encoder can yield strong general semantic representations, its features may contain structural artifacts such as normal object contours and texture boundaries. When such artifacts are preserved in the residuals, the anomaly segmentation network is prone to misclassifying normal structures as defects. If two separate encoders are used to perform retrieval and reconstruction independently, it will increase redundant computation and may cause semantic or spatial inconsistencies between the two feature paths.

[0006] In terms of residual calculation, cosine residuals based on element-wise product of normalized features have the advantages of fast calculation speed and easy optimization by graphics processors, but may retain more normal object structure; L1 residuals based on feature subtraction are more likely to highlight local anomalies, but may increase computational overhead.

[0007] Therefore, balancing the ability to distinguish anomalies in residuals with reasoning efficiency is a problem that urgently needs to be solved. Summary of the Invention

[0008] To overcome the existing technical problems, this invention provides an anomaly detection and localization method and system based on an adaptive residual algorithm.

[0009] The present invention adopts the following technical solution.

[0010] An anomaly detection and localization method based on an adaptive residual algorithm includes the following steps: S1. Acquire the image to be detected and perform preprocessing; S2. Extract the category label of the image to be detected, as well as multiple main image block features and auxiliary image block features, through dual encoders, and generate fusion query features; S3. Calculate the first similarity with each category prototype in the preset category prototype library based on the category label, and select multiple candidate main prototypes and multiple candidate auxiliary prototypes. Integrate the candidate main prototypes to obtain the candidate main prototype library. S4. Update the candidate master prototype library based on the master image patch features to obtain the dynamic master prototype library. Generate reconstruction weights based on the master image patch features and the dynamic master prototype library. Reconstruct the dynamic master prototype library and candidate auxiliary prototypes based on the reconstruction weights to obtain the reconstruction master features and reconstruction auxiliary features, and generate the fused reconstruction features. S5. Generate a hybrid residual based on the fusion query features and fusion reconstruction features; S6. Input the hybrid residual into the lightweight segmentation head to generate a pixel-level anomaly map so that anomaly determination and defect location can be output.

[0011] As a further improvement of the present invention, the preprocessing includes one or more of the following: size normalization, region clipping, numerical normalization, and data augmentation.

[0012] As a further improvement of the present invention, the preset category prototype library has a corresponding main prototype library and auxiliary prototype library. Each static main prototype in the main prototype library, each auxiliary prototype in the auxiliary prototype library, and each category prototype in the category prototype library are all in one-to-one correspondence through an index. The specific steps for selecting multiple candidate main prototypes and multiple candidate auxiliary prototypes include: selecting a preset number of indexes based on the first similarity between the category label and the prototype of each category from high to low to form a candidate index set; obtaining the corresponding static main prototype from the main prototype library based on the candidate index set to obtain multiple candidate main prototypes; and obtaining the corresponding auxiliary prototype from the auxiliary prototype library based on the candidate index set to obtain multiple candidate auxiliary prototypes.

[0013] As a further improvement of the present invention, the method for constructing the category prototype library, the main prototype library, and the auxiliary prototype library specifically includes: obtaining a set of normal reference images, preprocessing the normal reference images in the set of normal reference images, inputting each normal reference image into the main encoder of the dual encoder to obtain a reference category label and multiple reference main image block features, inputting each normal reference image into the auxiliary encoder of the dual encoder to obtain multiple reference auxiliary image block features, and making the reference main image block features and the reference auxiliary image block features correspond to each other in spatial location or sample index through alignment processing; Representative features are generated by aggregating the features of the reference master image block of at least one normal reference image, and the representative features are used as static master prototypes to form a master prototype library. The category prototypes of the corresponding static master prototype are obtained by aggregating the category labels of the normal reference image that generates any static master prototype, and a category prototype library is constructed based on all category prototypes. Based on the reference master image patch features that generate any static master prototype, select the corresponding reference auxiliary image patch features at the spatial location or sample index and aggregate them to obtain auxiliary prototypes. An auxiliary prototype library is constructed based on all auxiliary prototypes.

[0014] As a further improvement of the present invention, S4 specifically includes: S41. Input the main image block features into the learnable dynamic information extraction and mapping module to obtain dynamic information. Generate a compensation matrix that matches the candidate main prototype library through learnable projection or equivalent transformation. Combine the compensation matrix with the candidate main prototype library to obtain the dynamic main prototype library. S42. Calculate the cosine distance between the main image block features and the dynamic main prototypes in the dynamic main prototype library, and convert the cosine distance into reconstruction weights through monotonic transformation and normalization. S43. By reconstructing the weights, a weighted sum is performed on multiple dynamic master prototypes to obtain the reconstructed master features; S44. By reconstructing the weights, the candidate auxiliary prototypes corresponding to the dynamic main prototype are weighted and summed to obtain the reconstructed auxiliary features. S45. Input the reconstruction auxiliary features into the weak initialization gated mapping to obtain the adjustment coefficients, adjust the reconstruction auxiliary features according to the adjustment coefficients, and fuse them with the reconstruction main features to obtain the fused reconstruction features.

[0015] As a further improvement to the present invention, the specific steps of S5 include: S51. Restore the fused query features and fused reconstructed features to query feature tensors and reconstructed feature tensors with spatial structures respectively, and perform normalization to obtain the corresponding normalized query features and normalized reconstructed features. Calculate the product-type cosine residuals based on the normalized query features and normalized reconstructed features to form the basic residuals. S52. Calculate the cosine similarity of the normalized query feature and the normalized reconstructed feature for each spatial location to obtain the second similarity of the corresponding spatial location. Integrate the second similarity to obtain a two-dimensional similarity map. Copy the two-dimensional similarity map along the channel dimension to obtain a similarity tensor with the same shape as the basic residual. S53. Modulate the basic residuals according to the similarity tensor to obtain the hybrid residuals.

[0016] As a further improvement of the present invention, the lightweight segmentation head sequentially includes 1×1 convolution, batch normalization, ReLU activation, two-stage 4×4 transposed convolution batch normalization ReLU activation operations, output 1×1 convolution, and Sigmoid activation.

[0017] As a further improvement of the present invention, the training method for the parameters of the dynamic information extraction mapping module, the generation compensation matrix, and the weakly initialized gated mapping specifically includes: Normal images are used as images to be detected. S1 to S4 are performed to generate fusion training query features and fusion reconstruction features. The first-stage loss value is calculated using the mean cosine distance. By minimizing the first-stage loss value, the parameters of the dynamic information extraction mapping module, the compensation matrix, and the weakly initialized gated mapping are trained. The expression for the loss value in the first stage: , in, This is the first stage loss value. It is the total number of local feature locations in a single normal image. It is an index of the local feature location. It is the first of the fusion training query features The fused feature vector at each position, It is the first feature of fusion and reconstruction. The fused feature vector at each position.

[0018] Reference Figure 3 As can be seen, as a further improvement of the present invention, the training method for the parameters of the lightweight segmentation head specifically includes: acquiring a synthetic anomalous image and a corresponding synthetic anomalous mask; performing S1 to S6 to generate a pixel-level anomalous map as a predicted anomalous map based on the synthetic anomalous image as the image to be detected; calculating pixel difference loss and classification loss based on the predicted anomalous map and the synthetic anomalous mask to obtain a second-stage loss value; and training the parameters of the lightweight segmentation head by minimizing the second-stage loss value. The expression for calculating the loss value in the second stage: , in, This is the second-stage loss value. and It is loss weight. It is used to constrain the pixel differences between the predicted anomaly map and the synthesized anomaly mask. It is a classification loss used to enhance the discrimination ability of hard-to-classify pixels.

[0019] This invention also proposes an anomaly detection and localization system based on an adaptive residual algorithm, employing the above-mentioned anomaly detection and localization method based on an adaptive residual algorithm, including: The complementary feature extraction module is used to extract the category label, main image patch features and auxiliary image patch features of the image to be detected through the main encoder and auxiliary encoder in the dual encoder. The complementary prototype library building module is used to build the category prototype library, main prototype library, and auxiliary prototype library corresponding to the index; The category-aware retrieval module is used to calculate the first similarity between the category label of the image to be detected and each category prototype in the category prototype library, and select multiple corresponding static main prototypes in the main prototype library and multiple corresponding auxiliary prototypes in the auxiliary prototype library to obtain multiple candidate main prototypes and multiple candidate auxiliary prototypes. The dynamic prototype refinement module is used to generate a compensation matrix and update the dynamic master prototype library based on the main image patch features of the image to be detected. The complementary prototype reconstruction module is used to generate reconstruction weights and reconstruct the dynamic main prototype library and candidate auxiliary prototypes based on the reconstruction weights to obtain the reconstructed main features and reconstruction auxiliary features. The primary and secondary feature fusion module is used to generate fused query features and fused reconstructed features; The hybrid residual generation module is used to calculate the basic residuals of the fused query features and the fused reconstructed features and the second similarity of the corresponding spatial locations, and modulate the basic residuals according to the second similarity to obtain the hybrid residuals; The anomaly segmentation and scoring module is used to generate pixel-level anomaly maps based on the mixed residuals; The result output module is used to output a normal or abnormal judgment based on the pixel-level anomaly map, and output the defect location.

[0020] The beneficial effects of this invention are as follows: 1. Narrowing the scope of retrieval calculations and reducing the overhead of multi-category detection. By calculating the first similarity between category tags and a pre-defined category prototype library, candidate primary and auxiliary prototypes are selected, significantly reducing the computational load of subsequent steps. Furthermore, since the category prototype library, primary prototype library, and auxiliary prototype library are constructed through aggregation, storage and matching computation are greatly reduced, supporting multiple product categories sharing a single detection model.

[0021] 2. Dynamically adapt to the image to be detected to ensure reconstruction quality. A dynamic master prototype library is obtained by updating the candidate master prototype library based on the features of the master image patch, making the candidate master prototypes fit the image to be detected and improving the accuracy of reconstructed master features and auxiliary features. Reconstructed master features and auxiliary features are obtained based on the same reconstruction weight, avoiding repeated similarity calculations in auxiliary branches, while ensuring semantic and spatial alignment of the two reconstructed features.

[0022] 3. Dual encoder complementary representation to suppress false detections of normal structures. By extracting main image patch features and auxiliary image patch features through dual encoders, and fusing them to generate fused query features and fused reconstruction features, the complementary representation of the two features weakens the structural artifacts such as normal object contours and texture boundaries remaining in the single encoder features, thereby reducing abnormal false responses in normal areas.

[0023] 4. Similarity-modulated hybrid residuals balance accuracy and inference efficiency. The basic residuals of the fused query features and the fused reconstructed features, along with the second similarity of their corresponding spatial locations, are calculated. The hybrid residuals are obtained by modulating the basic residuals using the second similarity. This method is based on efficient product-type cosine residuals, while adaptively suppressing residuals in normal regions and highlighting differences in abnormal regions, thus improving anomaly discrimination capabilities while ensuring inference speed.

[0024] 5. Adaptable to industrial quality inspection scenarios. This invention only requires normal images for construction, reducing the dependence on real defect samples and pixel-level annotations, and adapting to production scenarios where industrial defects are low-frequency and diverse in form. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is the overall process flowchart of the present invention; Figure 3 This is a flowchart of the second stage training process of the present invention; Figure 4 This is a comparison diagram showing the effects of the present invention compared to existing technologies. Detailed Implementation

[0027] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions.

[0028] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings. The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] Reference Figures 1-4 Therefore, this invention proposes an anomaly detection and localization method based on an adaptive residual algorithm, comprising the following steps: S1. Acquire the image to be detected and perform preprocessing; As a further improvement of the present invention, the preprocessing includes one or more of the following: size normalization, region clipping, numerical normalization, and data augmentation.

[0030] S2. Extract the category label of the image to be detected, as well as multiple main image block features and auxiliary image block features, through dual encoders, and generate fusion query features; The dual encoder consists of a master encoder and an auxiliary encoder. The master encoder is a visual feature extraction network used to extract features from the main image patch, and can employ a pre-trained visual model with both global semantic and local representation capabilities. The auxiliary encoder is a visual feature extraction network used to extract auxiliary image patch features that are complementary to the main image patch features. Its training paradigm or network structure can differ from that of the master encoder. By leveraging the complementary representation of the two feature paths, structural artifacts such as normal object contours and texture boundaries that remain in the features of a single encoder are weakened, reducing abnormal false responses in normal regions.

[0031] Preferably, the main encoder and the auxiliary encoder employ different training paradigms or different network structures, enabling the auxiliary image patch features to supplement or correct structural artifacts that may exist in the main image patch features. It should be noted that the main encoder and auxiliary encoder can be replaced by other self-supervised models, supervised models, convolutional neural networks, visual Transformers, or multimodal encoders, and the number of auxiliary encoders can also be more than one.

[0032] Category labels are feature vectors used to characterize the global semantics of the image to be detected, and are used to retrieve candidate primary and candidate auxiliary prototypes related to the image to be detected from the category prototype library.

[0033] Main image patch features and auxiliary image patch features, also known as image patch labels, are used to characterize the feature vectors of local regions in the image to be detected. After dividing or mapping the image to be detected into multiple spatial locations, each spatial location corresponds to an image patch label.

[0034] The set of features of the main image patch is represented as Category tags are represented as The auxiliary image patch feature set is represented as ,in, This represents the number of local feature locations. Number of channels in the main prototype The auxiliary feature channel number is used. Of course, when it is necessary to unify the channel dimension, the auxiliary image patch features can be projected to the C dimension.

[0035] It should be further explained that when extracting the main image patch features and the auxiliary image patch features, the main image patch features and the auxiliary image patch features are aligned. The specific method of alignment is the same as that described in the construction method of the category prototype library, the main prototype library and the auxiliary prototype library below.

[0036] S3. Calculate the first similarity with each category prototype in the preset category prototype library based on the category label, and select multiple candidate main prototypes and multiple candidate auxiliary prototypes. Integrate the candidate main prototypes to obtain the candidate main prototype library. Specifically, the first similarity is cosine similarity, which is used for the first similarity in the category prototype library. Each category prototype Calculate its relationship with the category label The cosine similarity is calculated using the following expression: , in, It is the first similarity. It is a category tag. It is the first in the category prototype library Each category prototype This refers to the number of static master prototypes, which is described in detail in the steps of building the category prototype library below.

[0037] As a further improvement of the present invention, the preset category prototype library has a corresponding main prototype library and auxiliary prototype library. Each static main prototype in the main prototype library, each auxiliary prototype in the auxiliary prototype library, and each category prototype in the category prototype library are all in one-to-one correspondence through an index. The specific steps for selecting multiple candidate main prototypes and multiple candidate auxiliary prototypes include: selecting a preset number of indexes based on the first similarity between the category label and the prototype of each category from high to low to form a candidate index set; obtaining the corresponding static main prototype from the main prototype library based on the candidate index set to obtain multiple candidate main prototypes; and obtaining the corresponding auxiliary prototype from the auxiliary prototype library based on the candidate index set to obtain multiple candidate auxiliary prototypes.

[0038] This step is performed in the dynamic prototype adapter, i.e. Figure 2 The process is shown in the lower left corner. In this invention, the preferred method for selecting a preset number of candidate indices is Top-K. The top K candidate objects are selected from highest to lowest similarity, where K is a positive integer and is necessarily much smaller than the capacity Z of the main prototype library. Subsequent dynamic adjustments and reconstructions are only performed on a small number of related prototypes, thereby reducing the impact of irrelevant product categories and lowering the computational load.

[0039] The expression for the candidate index set: , Expression for the candidate main prototype library: , in, This refers to the value of K in the Top-K method, and C is the number of main prototype channels, which is described in detail in the steps of building the category prototype library below. It is a set of candidate indices. This is the expression for the main prototype library, which is described in detail in the steps of building the category prototype library below.

[0040] Furthermore, cosine similarity and Top-K retrieval are preferred embodiments of this invention. This invention can also replace cosine similarity with Euclidean distance, Mahalanobis distance, dot product, or learnable similarity. Similarly, similarity thresholds, approximate nearest neighbor indexes, hierarchical retrieval, or product category priors can also replace fixed Top-K retrieval.

[0041] In existing technologies, prototype libraries store a large number of repetitive or similar image patch features. The large size of these libraries leads to a decrease in retrieval speed as the number of samples and product categories increases. Furthermore, when multiple product categories share a model, directly matching within the entire prototype library can cause irrelevant category prototypes to expand the search space and potentially interfere with reconstruction.

[0042] Therefore, as a further improvement of the present invention, the method for constructing the category prototype library, the main prototype library, and the auxiliary prototype library specifically includes: obtaining a set of normal reference images, preprocessing the normal reference images in the set of normal reference images, inputting each normal reference image into the main encoder of the dual encoder to obtain a reference category label and multiple reference main image block features, inputting each normal reference image into the auxiliary encoder of the dual encoder to obtain multiple reference auxiliary image block features, and making the reference main image block features and the reference auxiliary image block features correspond to each other in spatial location or sample index through alignment processing; Furthermore, as a preferred embodiment of the present invention, before obtaining the normal reference image set, the method further includes: obtaining a normal training image set consisting of G product categories. Each category Only defect-free normal reference images are included. All or a portion of the normal reference images can be selected from this set of normal training images to form the normal reference image set. .

[0043] Similarly, the preprocessing performed on the normal reference image here is the same as the preprocessing method performed on the image to be detected in S1, which includes size normalization, region cropping, numerical normalization, and optional data augmentation to ensure that the input meets the spatial size and numerical range requirements of the main encoder and auxiliary encoder. Specific input resolution, cropping ratio, interpolation method, and augmentation parameters are engineering parameters that can be configured according to the device and dataset.

[0044] Alignment processing can specifically involve interpolation, pooling, feature layer selection, or coordinate mapping to ensure that the features of the reference main image patch and the reference auxiliary image patch correspond to each other in spatial location or sample index. If necessary, linear projection, convolutional projection, or channel selection can be used to transform the two feature paths to a suitable channel dimension for fusion.

[0045] Representative features are generated by aggregating the features of the reference master image block of at least one normal reference image, and the representative features are used as static master prototypes to form a master prototype library. Specifically, a candidate feature set is obtained from the reference master image patch features of the normal reference image, and redundancy is reduced through clustering, representativeness screening, or learnable quantization. The obtained representative features constitute the master prototype library.

[0046] in, This represents the number of static master prototypes. The number of channels in the main prototype. Determined based on normal sample size, product category, storage resources, and inference latency constraints.

[0047] Clustering, specifically, is a common image feature encoding method. Traditional methods like the Bag of Visual Words (BoVW) and VLAD (Vector of Locally Aggregated Descriptors) encode image features using k-means clustering. Subsequent recognition is then based directly on these encodings rather than the image features themselves.

[0048] The category prototypes of the corresponding static master prototype are obtained by aggregating the category labels of the normal reference image that generates any static master prototype, and a category prototype library is constructed based on all category prototypes. Based on the reference master image patch features that generate any static master prototype, select the corresponding reference auxiliary image patch features at the spatial location or sample index and aggregate them to obtain auxiliary prototypes. An auxiliary prototype library is constructed based on all auxiliary prototypes.

[0049] This leads to:

[0050] in, For category prototype library, To assist the prototype library, This is the number of auxiliary feature channels. If a unified channel dimension is needed later, the auxiliary prototype can be projected onto... dimension.

[0051] It should be noted that the "prototype" in the static primary prototype, category prototype, and auxiliary prototype of this invention refers to a representative vector used to represent a class or cluster of normal visual features. A prototype can be a cluster center, a representative feature, a learnable vector, or a feature vector obtained by aggregating multiple normal features. The above technical solution is only a preferred embodiment of this invention.

[0052] The category prototype library, main prototype library, and auxiliary prototype library constitute a compact, complementary prototype library. Corresponding elements in these three libraries are linked through the same index, allowing the retrieval of corresponding static main prototypes, category prototypes, and auxiliary prototypes using the same search index. That is, the category prototype... Static main prototype and auxiliary prototype Together, they describe a normal semantic or local pattern. The index alignment relationship can be directly saved as array positions, or as a mapping table, key-value pairs, or database association fields.

[0053] It should be noted that the construction of the complementary prototype library of the present invention can be achieved through core set screening, online updating, vector quantization, or other methods that can reduce the redundancy of normal features, and is not limited to the above-mentioned technical solutions.

[0054] S4. Update the candidate master prototype library based on the master image patch features to obtain the dynamic master prototype library. Generate reconstruction weights based on the master image patch features and the dynamic master prototype library. Reconstruct the dynamic master prototype library and candidate auxiliary prototypes based on the reconstruction weights to obtain the reconstruction master features and reconstruction auxiliary features, and generate the fused reconstruction features. If a fixed prototype is used, it cannot adaptively adjust based on the category semantics, pose, texture, and local structure of the current image to be detected. Therefore, this invention updates the candidate master prototype library.

[0055] As a further improvement of the present invention, S4 specifically includes: S41. Input the main image block features into the learnable dynamic information extraction and mapping module to obtain dynamic information. Generate a compensation matrix that matches the candidate main prototype library through learnable projection or equivalent transformation. Combine the compensation matrix with the candidate main prototype library to obtain the dynamic main prototype library. The expression that yields dynamic information: , in, It is dynamic information. The learnable parameters representing the dynamic information extraction mapping. It is a dynamic information extraction function.

[0056] Preferably, the expression for the compensation matrix is ​​obtained as follows: , The expression for obtaining the dynamic master prototype library: , in, This is a candidate master prototype library, and the above addition represents the preferred residual prototype update method. Its technical meaning is that the candidate master prototype provides a basic normal pattern from a normal reference image, and the compensation matrix provides conditional information extracted from the current image to be detected, thus enabling the dynamic master prototype to maintain normal prototype constraints while adapting to the category, pose, texture, or local structure of the current image.

[0057] It should be noted that, without changing the above functions, dynamic information extraction can be achieved by attention, pooling and nonlinear mapping, convolutional mapping or gating networks; prototype updates can be achieved by addition, scaling, affine transformation, gating combination, low-rank update or other conditional update methods.

[0058] S42. Calculate the cosine distance between the main image block features and the dynamic main prototypes in the dynamic main prototype library, and convert the cosine distance into reconstruction weights through monotonic transformation and normalization. The expression for calculating cosine distance: , in, It is the first Features of each main image patch and the A dynamic master prototype The cosine distance.

[0059] The expression for converting to reconstructed weights is as follows: , in, Let be the weight transformation function that decreases with increasing distance. `Normalize` represents a function that normalizes the weights at the same location according to a predetermined normalization condition. Finally, the weights for all spatial locations form the reconstructed weight matrix. It should be noted that N refers to the number of local feature locations. It represents the number of K values ​​in a Top-K search.

[0060] S43. By reconstructing the weights, a weighted sum is performed on multiple dynamic master prototypes to obtain the reconstructed master features; The expression for reconstructing the main feature: , This soft reconstruction method allows each spatial location of the image to be detected to be represented by multiple related normal prototypes, rather than simply copying a single nearest neighbor feature, thereby improving the continuity of reconstruction and adaptability to changes in normal patterns.

[0061] In different implementations, exponential normalization, kernel functions, sparse normalization, learnable weight mapping, or finite nearest neighbor weighting can be used.

[0062] S44. By reconstructing the weights, the candidate auxiliary prototypes corresponding to the dynamic main prototype are weighted and summed to obtain the reconstructed auxiliary features. The expression for reconstructing auxiliary features: , Since the primary and secondary prototypes correspond to each other in the index, the weight of the j-th dynamic primary prototype in the main branch can be used for its corresponding j-th candidate secondary prototype. This approach maintains the semantic and spatial correspondence between the two reconstructed features while avoiding repeated N×K similarity calculations and weight normalizations in the secondary branches.

[0063] Of course, the above technical solution is a preferred solution of the present invention. In addition to complete sharing, the weight sharing can also be used for auxiliary branches after temperature calibration, linear calibration or gating calibration of the shared weights. Alternatively, the shared weights and the independent weights of the auxiliary branches can be combined according to a predetermined or learnable ratio.

[0064] The candidate auxiliary prototypes of the present invention share the reconstruction weights of the dynamic main prototype, which can avoid the repeated calculation of cosine distance and reconstruction weights and the possible disruption of the correspondence between the main and auxiliary features.

[0065] S45. Input the reconstruction auxiliary features into the weak initialization gated mapping to obtain the adjustment coefficients, adjust the reconstruction auxiliary features according to the adjustment coefficients, and fuse them with the reconstruction main features to obtain the fused reconstruction features.

[0066] It should be noted that the principles of the fusion steps for fusion reconstruction features and fusion query features are completely the same. Therefore, this invention will not elaborate on the specific fusion of fusion query features.

[0067] Expression for fusing and reconstructing features: , in, _ represents element-wise multiplication, _Gate represents controlled gating mapping, and _Fuse represents channel concatenation, weighted addition, or other fusion operations that preserve spatial correspondence.

[0068] Expressions that combine query features: , Weak initialization gating controls the amount of auxiliary branch injection during the initial training phase, allowing the model to learn the contribution of auxiliary information at each position or channel based on the training data from the first phase.

[0069] As a further improvement of the present invention, the training method for the parameters of the dynamic information extraction mapping module, the generation compensation matrix, and the weakly initialized gated mapping specifically includes: Normal images are used as images to be detected. S1 to S4 are performed to generate fusion training query features and fusion reconstruction features. The first-stage loss value is calculated using the mean cosine distance. By minimizing the first-stage loss value, the parameters of the dynamic information extraction mapping module, the compensation matrix, and the weakly initialized gated mapping are trained. The expression for the loss value in the first stage: , in, This is the first stage loss value. It is the total number of local feature locations in a single normal image. It is an index of the local feature location. It is the first of the fusion training query features The fused feature vector at each position, It is the first feature of fusion and reconstruction. The fused feature vector at each position.

[0070] The first stage uses normal images as training samples, eliminating the need for real outlier samples. Minimizing the first-stage loss allows the query features of normal images to be reconstructed with high quality from the dynamic master prototype. It's important to note that during the first-stage training, the master encoder, auxiliary encoder, and master prototype library can be frozen according to implementation requirements.

[0071] S5. Generate hybrid residuals based on the fusion query features and fusion reconstruction features.

[0072] More specifically, step S5 is to calculate the basic residuals of the fused query features and the fused reconstructed features and the second similarity of the corresponding spatial locations, and modulate the basic residuals according to the second similarity to obtain the mixed residuals; Using traditional cosine residuals can easily preserve the normal object structure, while directly using L1 interpolation may reduce inference efficiency.

[0073] As a further improvement to the present invention, the specific steps of S5 include: S51. Restore the fused query features and fused reconstructed features to query feature tensors and reconstructed feature tensors with spatial structures respectively, and perform normalization to obtain the corresponding normalized query features and normalized reconstructed features. Calculate the product-type cosine residuals based on the normalized query features and normalized reconstructed features to form the basic residuals. Regarding the specific steps of the recovery, as a concrete example of the present invention, one feature corresponds to a small region of the image. If the image contains 10x10 small regions, and each region has a d-dimensional feature, then the image has 100 d-dimensional features. This means that these 100 features are arranged into a 10x10xd tensor.

[0074] Specifically, normalization refers to performing L2 normalization along the channel dimension. Normalized query features and normalized reconstructed features can be represented as follows: .

[0075] The expression for calculating the product-type cosine residual is as follows: .

[0076] S52. Calculate the cosine similarity of the normalized query feature and the normalized reconstructed feature for each spatial location to obtain the second similarity of the corresponding spatial location. Integrate the second similarity to obtain a two-dimensional similarity map. Copy the two-dimensional similarity map along the channel dimension to obtain a similarity tensor with the same shape as the basic residual. S53. Modulate the basic residuals according to the similarity tensor to obtain the hybrid residuals.

[0077] The expression for the mixed residual: , For normal locations where the fused query features and fused reconstruction features are highly similar As the value approaches 1, the modulation term causes the mixed residual to approach a smaller value; for outliers where the two differ significantly, Deviations from 1 indicate that abnormal differences are preserved or enhanced. This approach primarily utilizes operations suitable for parallel computation, such as normalization, inner product, and element-wise multiplication, which can improve the discriminative power between normal and abnormal regions while avoiding the direct computation of high-overhead L1 feature differences.

[0078] Similarity modulation is not limited to the specific algebraic forms mentioned above. Any alternative implementation can be achieved by suppressing or enhancing the underlying residuals based on the similarity, distance, or confidence of the query features and reconstructed features in their corresponding spatial locations, in order to reduce normal structural responses and highlight anomalous differences.

[0079] S6. Input the hybrid residual into the lightweight segmentation head to generate a pixel-level anomaly map so that anomaly determination and defect location can be output.

[0080] Specifically, a pixel-level anomaly map is a pixel-level anomaly probability map. The values ​​in the pixel-level anomaly map represent the probability or degree of anomaly at the corresponding location, and the expression is: , It is a mixed residual, and D is a lightweight segmentation head.

[0081] In addition to generating pixel-level anomaly maps, this invention can also generate image-level anomaly scores. The anomaly detection system can be obtained based on the maximum value of a pixel-level anomaly image, the average of several maximum values, a predetermined quantile, or an image-level classification head. By comparing the pixel-level anomaly image or image-level anomaly score with a preset threshold, a normal / abnormal judgment can be output. Simultaneously, an anomaly heatmap can be overlaid onto the original image to output the defect location, area, outline, or alarm information. The preset threshold can be set according to actual needs, such as adjusting the threshold based on yield requirements.

[0082] The lightweight segmentation head is a network structure that converts mixed residuals into pixel-level anomaly maps, with fewer parameters and less computational cost than the main and auxiliary encoders.

[0083] Reference Figure 3 As can be seen, the lightweight segmentation head sequentially includes 1×1 convolution, batch normalization, ReLU activation, two-stage 4×4 transposed convolution with batch normalization and ReLU activation, output 1×1 convolution, and Sigmoid activation. It can also restore the anomaly probability map to a predetermined resolution through transposed convolution or interpolation, and can further restore it to the original image size.

[0084] The second phase of training will be conducted after the first phase of training.

[0085] Specifically, refer to Figure 3 As can be seen, the training method for the parameters of the lightweight segmentation head specifically includes: acquiring a synthetic anomalous image and the corresponding synthetic anomalous mask; using the synthetic anomalous image as the image to be detected, performing S1 to S6 to generate a pixel-level anomalous map as a predicted anomalous map; calculating the pixel difference loss and classification loss based on the predicted anomalous map and the synthetic anomalous mask to obtain the second-stage loss value; and training the parameters of the lightweight segmentation head by minimizing the second-stage loss value. The expression for calculating the loss value in the second stage: , in, This is the second-stage loss value. and It is loss weight. It is used to constrain the pixel differences between the predicted anomaly map and the synthesized anomaly mask. It is a classification loss used to enhance the discrimination ability of hard-to-classify pixels.

[0086] In one specific embodiment of the present invention, the loss weight =0.1, loss weight =0.5, an empirical value selected through experiments.

[0087] The first stage of training in this invention uses only normal images, while the second stage can use artificially synthesized anomalous samples for training. The segmentation head is trained using the sum of L1 segmentation loss and focal-based binary cross-entropy loss, without requiring the collection of real defect samples. Synthetic anomalous images can be obtained by applying external textures, noise, occlusion, color perturbations, or geometric perturbations to selected regions of the normal image, while simultaneously outputting a corresponding synthetic anomalous mask. Preferably, a product foreground mask can be generated by a pre-trained foreground segmentation model, ensuring that synthetic anomalous images are primarily located in the product area, reducing training bias caused by background synthetic anomalous images. Since techniques for distinguishing foregrounds are very common, this invention will not elaborate further.

[0088] Furthermore, by using a compressed complementary prototype library, a Top-K retrieval method, an auxiliary feature reconstruction weight sharing method, and a lightweight segmentation head, this invention can significantly reduce storage, retrieval, and network computation, making it suitable for online production line inspection and applicable to edge devices for real-time inspection.

[0089] This invention also proposes an anomaly detection and localization system based on an adaptive residual algorithm, employing the above-mentioned anomaly detection and localization method based on an adaptive residual algorithm, including: The complementary feature extraction module is used to extract the category label, main image patch features and auxiliary image patch features of the image to be detected through the main encoder and auxiliary encoder in the dual encoder. The complementary prototype library building module is used to build the category prototype library, main prototype library, and auxiliary prototype library corresponding to the index; The category-aware retrieval module is used to calculate the first similarity between the category label of the image to be detected and each category prototype in the category prototype library, and select multiple corresponding static main prototypes in the main prototype library and multiple corresponding auxiliary prototypes in the auxiliary prototype library to obtain multiple candidate main prototypes and multiple candidate auxiliary prototypes. The dynamic prototype refinement module is used to generate a compensation matrix and update the dynamic master prototype library based on the main image patch features of the image to be detected. The complementary prototype reconstruction module is used to generate reconstruction weights and reconstruct the dynamic main prototype library and candidate auxiliary prototypes based on the reconstruction weights to obtain the reconstructed main features and reconstruction auxiliary features. The primary and secondary feature fusion module is used to generate fused query features and fused reconstructed features; The hybrid residual generation module is used to calculate the basic residuals of the fused query features and the fused reconstructed features and the second similarity of the corresponding spatial locations, and modulate the basic residuals according to the second similarity to obtain the hybrid residuals; The anomaly segmentation and scoring module is used to generate pixel-level anomaly maps based on the mixed residuals; The result output module is used to output a normal or abnormal judgment based on the pixel-level anomaly map, and output the defect location.

[0090] The above modules can be implemented by software, hardware, firmware, or a combination thereof. Each module can be deployed on the same device or distributed across the image acquisition end, edge computing end, and server end.

[0091] Reference Figure 4 The performance comparison chart shows the accuracy of our method compared to all the latest methods in academia on the three major datasets: MVTec-AD, MVTec-3D, and Real-IAD. The best and second-best results are marked in red and blue, respectively. The algorithms, from top to bottom, are: RD4AD, UniAD, SimpleNet, DeSTSeg, DiAD, MambaAD, CPR, Dinomaly, INP-Former, WeakREST, and the algorithm of this invention. The datasets, from left to right, are MVTec-AD, MVTec-3D, and Real-IAD, with the last column representing the average value. Each dataset is further divided into four metrics from left to right: I-AUROC, P-AP, P-PRO, and P-AUROC. Furthermore, the average value is calculated as the average of all 12 metrics (4 metrics × 3 datasets). (-) indicates that some methods have missing values.

[0092] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An anomaly detection and localization method based on an adaptive residual algorithm, characterized in that, Includes the following steps: S1. Acquire the image to be detected and perform preprocessing; S2. Extract the category label of the image to be detected, as well as multiple main image block features and auxiliary image block features, through dual encoders, and generate fusion query features; S3. Calculate the first similarity with each category prototype in the preset category prototype library based on the category label, and select multiple candidate main prototypes and multiple candidate auxiliary prototypes. Integrate the candidate main prototypes to obtain the candidate main prototype library. S4. Update the candidate master prototype library according to the master image block features to obtain a dynamic master prototype library, generate reconstruction weights according to the master image block features and the dynamic master prototype library, reconstruct the dynamic master prototype library and candidate auxiliary prototypes according to the reconstruction weights to obtain reconstruction master features and reconstruction auxiliary features, and generate fused reconstruction features. S5. Generate a hybrid residual based on the fusion query features and fusion reconstruction features; S6. Input the hybrid residual into the lightweight segmentation head to generate a pixel-level anomaly map so that anomaly determination and defect location can be output.

2. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, The preprocessing includes one or more of the following: size normalization, region clipping, numerical normalization, and data augmentation.

3. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, The preset category prototype library has a corresponding main prototype library and auxiliary prototype library. Each static main prototype in the main prototype library, each auxiliary prototype in the auxiliary prototype library, and each category prototype in the category prototype library are all linked one-to-one by an index. The specific steps for selecting multiple candidate main prototypes and multiple candidate auxiliary prototypes include: selecting a preset number of indexes based on the first similarity between the category label and each category prototype from high to low to form a candidate index set; obtaining the corresponding static main prototype from the main prototype library based on the candidate index set to obtain multiple candidate main prototypes; and obtaining the corresponding auxiliary prototype from the auxiliary prototype library based on the candidate index set to obtain multiple candidate auxiliary prototypes.

4. The anomaly detection and localization method based on adaptive residual algorithm according to claim 3, characterized in that, The method for constructing the category prototype library, the main prototype library, and the auxiliary prototype library specifically includes: obtaining a set of normal reference images; preprocessing the normal reference images in the set of normal reference images; inputting each normal reference image into the main encoder of the dual encoder to obtain a reference category label and multiple reference main image block features; inputting each normal reference image into the auxiliary encoder of the dual encoder to obtain multiple reference auxiliary image block features; and aligning the reference main image block features and the reference auxiliary image block features to correspond to each other in spatial location or sample index. Representative features are generated by aggregating the features of the reference master image block based on at least one normal reference image, and the representative features are used as the static master prototype to constitute the master prototype library. The category prototypes corresponding to the static master prototype are obtained by aggregating the category tags of the normal reference image that generates any one of the static master prototypes, and the category prototype library is constructed based on all the category prototypes. Based on the reference master image patch features that generate any one of the static master prototypes, the corresponding reference auxiliary image patch features at spatial locations or sample indices are selected and aggregated to obtain auxiliary prototypes, and the auxiliary prototype library is constructed based on all the auxiliary prototypes.

5. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, S4 specifically includes: S41. Input the main image block features into the learnable dynamic information extraction and mapping module to obtain dynamic information. Generate a compensation matrix that is adapted to the candidate main prototype library through learnable projection or equivalent transformation. Combine the compensation matrix with the candidate main prototype library to obtain the dynamic main prototype library. S42. Calculate the cosine distance between the main image block features and the dynamic main prototypes in the dynamic main prototype library, and convert the cosine distance into reconstruction weights through monotonic transformation and normalization. S43. By reconstructing the weights, a weighted sum is performed on multiple dynamic master prototypes to obtain the reconstructed master features; S44. By reconstructing the weights, a weighted sum is performed on multiple candidate auxiliary prototypes corresponding to the dynamic main prototype to obtain the reconstructed auxiliary features; S45. Input the reconstruction auxiliary features into the weak initialization gating mapping to obtain the adjustment coefficients, adjust the reconstruction auxiliary features according to the adjustment coefficients, and fuse them with the reconstruction main features to obtain the fused reconstruction features.

6. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, The specific steps of S5 include: S51. Restore the fused query features and fused reconstruction features to query feature tensors and reconstruction feature tensors with spatial structures respectively, and perform normalization to obtain the corresponding normalized query features and normalized reconstruction features. Calculate the product-type cosine residuals based on the normalized query features and normalized reconstruction features to form the basic residuals. S52. Calculate the cosine similarity of the normalized query feature and the normalized reconstructed feature for each spatial location to obtain the second similarity of the corresponding spatial location. Integrate the second similarity to obtain a two-dimensional similarity map. Copy the two-dimensional similarity map along the channel dimension to obtain a similarity tensor with the same shape as the basic residual. S53. Modulate the basic residual according to the similarity tensor to obtain the mixed residual.

7. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, The lightweight segmentation head sequentially includes 1×1 convolution, batch normalization, ReLU activation, two-stage 4×4 transposed convolution batch normalization ReLU activation operations, output 1×1 convolution, and Sigmoid activation.

8. The anomaly detection and localization method based on adaptive residual algorithm according to claim 5, characterized in that, The training method for the dynamic information extraction mapping module, the generation of the compensation matrix, and the parameters of the weakly initialized gated mapping specifically includes: Using a normal image as the image to be detected, S1 to S4 are executed to generate fusion training query features and fusion reconstruction features. The first-stage loss value is calculated using the mean cosine distance. By minimizing the first-stage loss value, the parameters of the dynamic information extraction mapping module are trained, and the compensation matrix and the weakly initialized gated mapping are generated. The expression for the loss value in the first stage: ， in, This is the first stage loss value. It is the total number of local feature locations in a single normal image. It is an index of the local feature location. It is the first of the fusion training query features The fused feature vector at each position, It is the first feature of fusion and reconstruction. The fused feature vector at each position.

9. The anomaly detection and localization method based on adaptive residual algorithm according to claim 1, characterized in that, The training method for the parameters of the lightweight segmentation head specifically includes: acquiring a synthetic anomalous image and a corresponding synthetic anomalous mask; generating a pixel-level anomalous map as a predicted anomalous map based on the synthetic anomalous image as the image to be detected; calculating pixel difference loss and classification loss based on the predicted anomalous map and the synthetic anomalous mask to obtain a second-stage loss value; and training the parameters of the lightweight segmentation head by minimizing the second-stage loss value. The expression for calculating the loss value in the second stage: ， in, This is the second-stage loss value. and It is loss weight. It is used to constrain the pixel differences between the predicted anomaly map and the synthesized anomaly mask. It is a classification loss used to enhance the discrimination ability of hard-to-classify pixels.

10. An anomaly detection and localization system based on an adaptive residual algorithm, characterized in that, An anomaly detection and localization method based on an adaptive residual algorithm as described in any one of claims 1-9 includes: The complementary feature extraction module is used to extract the category label, main image patch features and auxiliary image patch features of the image to be detected through the main encoder and auxiliary encoder in the dual encoder. The complementary prototype library building module is used to build the category prototype library, main prototype library, and auxiliary prototype library corresponding to the index; The category-aware retrieval module is used to calculate the first similarity with each category prototype in the category prototype library based on the category label of the image to be detected, and select multiple corresponding static main prototypes in the main prototype library and multiple corresponding auxiliary prototypes in the auxiliary prototype library to obtain multiple candidate main prototypes and multiple candidate auxiliary prototypes. The dynamic prototype refinement module is used to generate a compensation matrix and update the dynamic master prototype library based on the main image block features of the image to be detected. The complementary prototype reconstruction module is used to generate reconstruction weights and reconstruct the dynamic main prototype library and candidate auxiliary prototypes according to the reconstruction weights to obtain the reconstructed main features and reconstruction auxiliary features. The primary and secondary feature fusion module is used to generate fused query features and fused reconstructed features; The hybrid residual generation module is used to calculate the basic residuals of the fused query features and the fused reconstructed features and the second similarity of the corresponding spatial locations, and modulate the basic residuals according to the second similarity to obtain the hybrid residuals; An anomaly segmentation and scoring module is used to generate pixel-level anomaly maps based on the hybrid residuals; The result output module is used to output a normal or abnormal judgment based on the pixel-level anomaly map, and output the defect location.

Images