Multi-modal priori knowledge fusion-based few-sample industrial anomaly detection method

Abstract

The invention discloses a few-sample industrial anomaly detection method based on multi-modal priori knowledge fusion, and belongs to the field of industrial anomaly detection in computer vision. The problems that a traditional anomaly detection method cannot fully consider subtle differences between different samples and lacks effective utilization of global information are solved. The invention introduces an innovative vision-language prior fusion method, and the core of the method lies in two main innovations: constructing a vision-language prior fusion module and an intra-class balance module. Through effective utilization of image-level reference features and block-level reference feature maps corresponding to vision-language information and reference sample images, two core innovation components of vision-language prior fusion and intra-class feature balance are introduced, vision prior information and language prior information are fused, intra-class changes are balanced, and the visual-language prior fusion and intra-class feature balance are realized. And more accurate industrial anomaly detection is realized. The method is mainly applied to industrial anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of industrial anomaly detection in computer vision, and specifically relates to a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion. Background Art

[0002] In the industrial manufacturing process, industrial anomaly detection is a key link in ensuring production quality. It not only directly affects the company's cost control and market reputation, but also determines whether the product can maintain its advantage in market competition. By discovering defects as early as possible in the production stage, industrial anomaly detection can effectively prevent unqualified products from entering the market, thereby reducing production losses. However, traditional industrial visual inspection relies on manual operation, which is time-consuming, labor-intensive, and inefficient, and it is difficult to adapt to the needs of modern industrial production for efficient detection.

[0003] In recent years, industrial anomaly detection has gradually become a research hotspot, with the goal of automatically identifying and locating anomalies in industrial product images through algorithms. However, existing methods face challenges such as lack of negative samples, uneven sample distribution, or vague defect definition, which makes it more difficult to build classifiers. In particular, when there are only "normal" samples, the accuracy of demarcating category boundaries is limited. To address these problems, researchers initially adopted one-class classification (OCC) methods, but this approach, which requires training models for different objects separately, is difficult to meet the complex and changing industrial needs. In contrast, the introduction of multi-class classification and a unified framework has significantly improved the detection efficiency. The core idea is to measure the difference between anomalies and normals through feature distance to achieve more robust anomaly detection.

[0004] With the development of pre-trained vision-language models such as CLIP, industrial anomaly detection methods have been further expanded. These models provide new ideas for anomaly detection with their excellent feature representation capabilities, especially in the task of object detection based on textual prompts. Under the vision-language paradigm, detection methods identify abnormal regions by measuring the distance between visual features and textual prompt features. However, compared with the actual needs of industrial scenarios, such methods still have room for improvement. Language prior methods tend to identify the most significant abnormal regions, such as highlighting only the larger parts of capsule cracks and ignoring smaller cracks. In contrast, visual prior methods are more aggressive and can identify the entire abnormal region, but may include normal areas (such as labels). This complementarity provides the possibility of integrating vision and language priors.

[0005] Despite this, current methods still face many limitations in few-sample scenarios. For example, when using multiple normal samples, the model is highly sensitive to background noise and position changes, and fails to fully consider subtle differences between samples. In addition, many methods treat reference samples equally and fail to give different samples differentiated weights, resulting in inaccurate detection results. At the same time, the nearest neighbor feature matching method based on the memory library pays more attention to local pixel features and lacks effective use of global information. These limitations show that how to efficiently integrate vision-vision and vision-language prior knowledge and mitigate the impact of intra-class changes while fusing memory library features is an urgent problem to be solved in current multimodal industrial anomaly detection research. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that traditional anomaly detection methods fail to fully consider the subtle differences between different samples and lack effective use of global information. The present invention provides a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion.

[0007] The few-sample industrial anomaly detection method based on multimodal prior knowledge fusion includes the following steps:

[0008] Training phase:

[0009] Construct a training sample set, wherein each training sample in the training sample set includes a sample image to be tested, and a prompt word and a true value segmentation map corresponding to the sample image to be tested, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample image to be tested in the training sample set includes a normal sample image and an abnormal sample image, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word;

[0010] The constructed visual-language prior fusion model is trained using each training sample. During the training process, the visual-language prior fusion model is used to output multiple image-level visual features F according to the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl , and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the visual-language prior fusion model are updated to complete the training; among them, the image-level visual feature F cls , Feature map of the spliced ​​blocks F p ' atch and language prior segmentation map M vl The number is the same, and there is a one-to-one correspondence;

[0011] Reasoning stage:

[0012] After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features F are output. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl ;

[0013] After summing and averaging all language prior segmentation maps, the generated standard segmentation map M v ' l sent to the adder;

[0014] K reference sample images are encoded by a pre-trained universal visual encoder to obtain multiple tile-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are sent to the constructed intra-class balancing module; K is an integer less than or equal to 4;

[0015] Through the constructed intra-class balancing module, each image-level visual feature F cls And its corresponding spliced ​​block feature map F p ' atch The image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image are compared and analyzed respectively, and the abnormal area location information map M corresponding to each image-level visual feature is output. mem ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all the block-level visual feature maps corresponding to each image-level visual feature;

[0016] After summing and averaging all abnormal area location information maps, the standard abnormal area location information map M′ is generated. mem Sent to the adder; the adder receives the standard segmentation map M v ' l and standard abnormal area location information map M′ mem After element-by-element addition, anomaly graph is generated to complete anomaly detection.

[0017] Preferably, the constructed vision-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module and a cosine similarity calculation module;

[0018] The pre-trained text encoder is used to encode the prompt word and obtain the text feature vector F t Send to the cosine similarity calculation module;

[0019] The pre-trained VLPF visual encoder is used to output multiple image-level visual features F according to the sample image to be tested. cls , and multiple tile-level visual feature maps F corresponding to each image-level visual feature patch , and the image-level visual feature F cls As the output of the vision-language prior fusion model;

[0020] The splicing module is used to combine multiple tile-level visual feature maps F corresponding to each image-level visual feature patch The height and width of the image are spliced ​​to obtain the spliced ​​block feature map F corresponding to each image-level visual feature. p ' atch , and the spliced ​​block feature map F p ' atch As the output of the vision-language prior fusion model;

[0021] Multi-layer perceptron, used to map the spliced ​​tile feature map F corresponding to each image-level visual feature p ' atch Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map F corresponding to each image-level visual feature p ″ atch ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model;

[0022] The visual prior information region generation module is used to generate the feature map F of each spliced ​​block. p ' atch Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature represents a spatial region with a length of N and a number of channels HW;

[0023] The first multiplier is used to multiply the dimensionally reduced block feature map F corresponding to each image-level visual feature. p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region ;

[0024] Average pooling module, used to pool the region-level visual features F region After pooling, the pooled region-level visual feature F is obtained r ' egion ;

[0025] The cosine similarity calculation module is used to calculate the text feature vector F corresponding to each image-level visual feature. t And the pooled region-level visual features F r ' egionPerform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature Represents a spatial region of length HW and channel number 1.

[0026] Preferably, the visual prior region R corresponding to each image-level visual feature is obtained. vpk The implementation method includes the following steps:

[0027] According to the feature map F of each spliced ​​block p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0028] Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map is a spatial region with a length of HW and a channel number of 1;

[0029] The visual prior score map S vpk Binarize and get the visual prior mask M vpk ;

[0030] For the visual prior mask M vpk Perform connected domain analysis to obtain the visual prior region R vpk .

[0031] Preferably, the intra-class balancing module outputs the abnormal region position information map M corresponding to each image-level visual feature. mem The implementation method includes the following steps:

[0032] The CTB module compares the image-level reference features corresponding to all reference samples with the image-level visual features F corresponding to each image-level visual feature. cls Perform class word balance to obtain the current image-level visual feature F cls The corresponding image-level weight

[0033]

[0034] The GPI module is used to map all tile-level reference feature maps corresponding to all reference samples and the current image-level visual feature F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Perform global prototype integration to obtain the prototype feature map F pro ;

[0035] Prototype feature map F pro And the spliced ​​block feature map F p' atch Perform cosine similarity calculation to obtain the anomaly score of the i-th block in the anomaly map

[0036] According to the abnormal scores of all blocks, the abnormal area location information map M is obtained. mem .

[0037] Preferably, the CTB module obtains the image-level weights The implementation methods include:

[0038] The image-level visual feature F corresponding to the current image-level visual feature cls The image-level features corresponding to the mth reference sample The cosine similarity of , and the reference weight of the mth reference sample is obtained

[0039] According to the reference weight Calculate the confidence weight of the mth reference sample

[0040]

[0041] in, Indicates the image-level feature corresponding to the mth reference sample Respectively Calculate the minimum value of cosine similarity, is the image-level feature corresponding to the t-th reference sample The cosine similarity of , m and t are the indexes of the reference samples, and m≠t;

[0042] Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight W cls , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight W ref ;

[0043] According to the total reference weight W cls and the total confidence weight W ref , get the image-level weight in,

[0044]

[0045] ‖·‖2 is used to calculate the L2 norm.

[0046] Preferably,

[0047] express and Calculate the cosine similarity. is the feature map of the spliced ​​block F p ' atch The features of the i-th block in , is the prototype feature map F pro The features of the i-th block in ;

[0048]

[0049] express With B patch After calculating the cosine similarity, take the largest value, B patch are all tile-level reference feature maps corresponding to all reference samples.

[0050] Preferably, the GPI module obtains the prototype feature map F pro The implementation methods include:

[0051] According to all the tile-level reference feature maps corresponding to all the reference samples and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Calculate the tile-level weight matrix W pro :

[0052]

[0053] <F p ' atch ,B patch > indicates F p ' atch With B patch Calculate the cosine similarity, B patch All tile-level reference feature maps corresponding to all reference samples;

[0054] For the tile-level weight matrix W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix

[0055]

[0056] according to and B patch , get the prototype feature map F pro , ‖·‖1 is used to calculate the L1 norm.

[0057] Preferably, the pre-trained VLPF visual encoder comprises an input layer, 24 self-attention encoding layers and an output layer connected in cascade sequence;

[0058] And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer;

[0059] Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features F cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch ;

[0060] The visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer. patch After extracting the features of the abnormal area, it is sent to the next layer;

[0061] The image-level visual features F output by the 6th, 12th, 18th, and 24th self-attention encoding layers cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch , as the output of the pre-trained VLPF visual encoder.

[0062] Preferably, the visual prior weighted attention module is used to overlay all tile-level visual feature maps F output by the previous layer. patch The specific process of extracting abnormal area features is as follows:

[0063] All the tile-level visual feature maps F output by the previous layer patch After splicing the height and width of the image, we get the spliced ​​block feature map F p ' atch Send it to the second multiplier, and at the same time, according to the spliced ​​block feature map F p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0064] Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map Represents a spatial region with a length of HW and a channel number of 1;

[0065] The visual prior score map S vpk Binarize and get the visual prior mask M vpk ;

[0066] The visual prior mask M vpk Pixels above the threshold 0 are assigned weight w1, and pixels below or equal to the threshold 0 are assigned weight w2. The resulting weighted visual prior mask Sent to the second multiplier;

[0067] The second multiplier converts F p ' atch and After element-wise multiplication, a tile-level weighted feature map is generated And split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is HW and the number of channels is C patch space region, The height is H, the width is W, and the number of channels is C patch spatial domain.

[0068] Preferably,

[0069] express and The first m in the cosine similarity matrix k The highest value, and They are the feature maps of the spliced ​​blocks F p ' atch In the i-th and j-th tile-level features in the height dimension, the values ​​of i and j range from 1 to HW.

[0070] Advantages of the present invention:

[0071] The present invention provides a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion. The method introduces an innovative vision-language prior fusion method. The core of this method lies in two major innovations: constructing a vision-language prior fusion module and an intra-class balancing module.

[0072] The constructed visual-language prior fusion module fuses the two modalities of vision and language, and utilizes language and visual information at the same time. Its design goal is to fuse visual prior information with language prior information, and has the function of fusing visual prior information into block-level feature maps, so that it can pay more attention to areas where anomalies may exist and ignore those areas where anomalies are unlikely to exist, thereby enhancing the accuracy of anomaly detection.

[0073] The constructed intra-class balancing module integrates comprehensive reference information to replace the traditional nearest neighbor search method to optimize the retrieval process of the memory library. Its goal is to identify the prototype pattern in the query feature. The intra-class balancing module reduces the interference of intra-class differences between multiple references by extracting the global features of a single reference, aiming to use the reference features more efficiently. It extracts the corresponding prototype feature map for each block-level feature map of the sample to be tested, and compares the relationship between the two to detect abnormal areas. At the same time, it balances the changes within the class through the image-level class features to improve the efficiency of the use of reference information. In addition, in order to balance the differences between different reference samples, the intra-class feature balance also comprehensively considers the changes in global features.

[0074] In general, the whole architecture combines visual principles and visual-text principles for industrial visual anomaly detection. It reduces intra-class variations in reference samples and integrates all features of the data in the memory library (that is, image-level reference features and tile-level reference feature maps corresponding to the reference sample images) to capture the global correlation of features, effectively solving the problem of insufficient consideration of image-level information, and providing new ideas and methods for visual-language detection in industrial anomaly detection.

[0075] The present invention introduces two core innovative components, namely, visual-language prior fusion and intra-class feature balance, by effectively utilizing visual-language information and data in the memory library (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images), thereby reducing the reliance on single information or traditional methods in the detection process, thereby improving the performance of the model in few-sample industrial anomaly detection and achieving more accurate industrial anomaly detection.

[0076] The present invention combines visual principles and visual-text principles for industrial visual anomaly detection. It reduces intra-class variations in reference samples and integrates all features of image-level reference features and tile-level reference feature maps corresponding to all reference sample images in the memory library to capture the global correlation of features, effectively solving the problem of insufficient consideration of image-level information. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the principle of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion in the training stage of the present invention;

[0078] Figure 2 : is a schematic diagram of the principle of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion in the reasoning stage of the present invention; wherein, is the image-level reference feature corresponding to the mth reference sample, are all tile-level reference feature maps corresponding to the m-th reference sample;

[0079] Figure 3It is a schematic diagram of the internal principle of the intra-class balancing module;

[0080] Figure 4 It is a schematic diagram of the internal structure of the VLPF visual encoder. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0083] This paper aims to enhance the model's ability to process and analyze relevant data in industrial anomaly detection, improve the accuracy and reliability of detection, and proposes a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion. This method optimizes the process of feature extraction and fusion in few-sample industrial anomaly detection from the perspective of multimodal information utilization and innovative component construction.

[0084] Specific implementation method 1. Combination Figure 1 and Figure 2 As shown, the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion described in this embodiment includes the following steps:

[0085] Training phase:

[0086] Construct a training sample set, wherein each training sample in the training sample set includes a sample image to be tested, and a prompt word and a true value segmentation map corresponding to the sample image to be tested, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample image to be tested in the training sample set includes a normal sample image and an abnormal sample image, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word;

[0087] The constructed visual-language prior fusion model is trained using each training sample. During the training process, the visual-language prior fusion model is used to output multiple image-level visual features F according to the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl, and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the visual-language prior fusion model are updated to complete the training; among them, the image-level visual feature F cls , Feature map of the spliced ​​blocks F p ' atch and language prior segmentation map M vl The number is the same, and there is a one-to-one correspondence;

[0088] Reasoning stage:

[0089] After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features F are output. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl ;

[0090] After summing and averaging all language prior segmentation maps, the generated standard segmentation map M v ' l sent to the adder;

[0091] K reference sample images are encoded by a pre-trained universal visual encoder to obtain multiple tile-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are sent to the constructed intra-class balancing module; K is an integer less than or equal to 4;

[0092] Through the constructed intra-class balancing module, each image-level visual feature F cls And its corresponding spliced ​​block feature map F p ' atch The image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image are compared and analyzed respectively, and the abnormal area location information map M corresponding to each image-level visual feature is output. mem ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all the block-level visual feature maps corresponding to each image-level visual feature;

[0093] After summing and averaging all abnormal area location information maps, the standard abnormal area location information map M′ is generated. mem sent to the adder;

[0094] Adder receives the standard partition diagram M v ' l and standard abnormal area location information map M′ mem After element-by-element addition, anomaly graph is generated to complete anomaly detection.

[0095] The core of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion described in this embodiment lies in two major innovations: constructing a visual-language prior fusion module and an intra-class balancing module.

[0096] The constructed visual-language prior fusion module fuses the two modalities of vision and language, and utilizes language and visual information at the same time. Its design goal is to fuse visual prior information with language prior information, and has the function of fusing visual prior information into block-level feature maps, so that it can pay more attention to areas where anomalies may exist and ignore those areas where anomalies are unlikely to exist, thereby enhancing the accuracy of anomaly detection.

[0097] The constructed intra-class balancing module can fully integrate the tile-level information of the reference sample into the tile-level prototype features, retaining the globality of the reference sample. At the same time, the integration of the tile-level prototype features is adjusted through the differences in the image-level features between the sample to be tested and the reference sample, and between the reference samples, so as to achieve more accurate anomaly positioning.

[0098] The constructed intra-class balancing module integrates comprehensive reference information to replace the traditional nearest neighbor search method to optimize the retrieval process of the memory library. Its goal is to identify the prototype pattern in the query feature. The intra-class balancing module reduces the interference of intra-class differences between multiple references by extracting the global features of a single reference, aiming to use the reference features more efficiently. It extracts the corresponding prototype feature map for each block-level feature map of the sample to be tested, and compares the relationship between the two to detect abnormal areas. At the same time, it balances the changes within the class through the image-level class features. Improve the efficiency of the use of reference information. In addition, in order to balance the differences between different reference samples, the intra-class feature balance also comprehensively considers the changes in global features.

[0099] In general, the whole architecture combines visual principles and visual-text principles for industrial visual anomaly detection. It reduces intra-class variations in reference samples and integrates all features of the memory data (i.e., image-level reference features and tile-level reference feature maps corresponding to the reference sample images) to capture the global correlation of features, effectively solving the problem of insufficient consideration of image-level information, and providing new ideas and methods for visual-language detection in industrial anomaly detection.

[0100] The present invention introduces two core innovative components, namely, visual-language prior fusion and intra-class feature balance, by effectively utilizing visual-language information and memory data (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images), thereby reducing the reliance on single information or traditional methods in the detection process, thereby improving the performance of the model in few-sample industrial anomaly detection and achieving more accurate industrial anomaly detection.

[0101] For details, see Figure 1,The constructed visual-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module and a cosine similarity calculation module;

[0102] The pre-trained text encoder is used to encode the prompt word and obtain the text feature vector F t Send to the cosine similarity calculation module;

[0103] The pre-trained VLPF visual encoder is used to output multiple image-level visual features F according to the sample image to be tested. cls , and multiple tile-level visual feature maps F corresponding to each image-level visual feature patch , and the image-level visual feature F cls As the output of the vision-language prior fusion model;

[0104] The splicing module is used to combine multiple tile-level visual feature maps F corresponding to each image-level visual feature patch The height and width of the image are spliced ​​to obtain the spliced ​​block feature map F corresponding to each image-level visual feature. p ' atch , and the spliced ​​block feature map F p ' atch As the output of the vision-language prior fusion model;

[0105] Multi-layer perceptron, used to map the spliced ​​tile feature map F corresponding to each image-level visual feature p ' atch Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map F corresponding to each image-level visual feature p ″ atch ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model;

[0106] The visual prior information region generation module is used to generate the feature map F of each spliced ​​block. p ' atch Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature represents a spatial region with a length of N and a number of channels HW;

[0107] The first multiplier is used to multiply the dimensionally reduced block feature map F corresponding to each image-level visual feature. p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region ; Specifically,

[0108]

[0109] m α is a hyperparameter, For R vpk The nth region in the length dimension, For R vpk The eigenvalue of the nth region and the i-th channel in the length dimension;

[0110] Average pooling module, used to pool the region-level visual features F region After pooling, the pooled region-level visual feature F is obtained r ' egion ;

[0111] The cosine similarity calculation module is used to calculate the text feature vector F corresponding to each image-level visual feature. t And the pooled region-level visual features F r ' egion Perform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature Represents a spatial region of length HW and channel number 1.

[0112] In this preferred embodiment, the constructed visual-language prior fusion model introduces regional-level visual features into the visual-language similarity calculation, which enhances the network's attention to potential abnormal areas. By integrating visual prior knowledge, the visual information pays more attention to areas where abnormalities may exist and ignores areas where abnormalities are unlikely to exist.

[0113] In order to align with the text features, a multi-layer perceptron (MLP) is used to Perform dimensionality reduction to generate post-processing features

[0114] See also Figure 1 , further, we get the visual prior region R corresponding to each image-level visual feature vpk The implementation method includes the following steps:

[0115] According to the feature map F of each spliced ​​block p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0116] Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map is a spatial region with a length of HW and a channel number of 1;

[0117] The visual prior score map Svpk Binarize and get the visual prior mask M vpk ;

[0118] For the visual prior mask M vpk Perform connected domain analysis to obtain the visual prior region R vpk .

[0119] In this preferred embodiment, attention is generated by using a visual priori mask and combined with connected region analysis to expand the mask from the tile level to the region level analysis, thereby achieving a higher level of feature understanding. vpk Apply connected region analysis to generate binary visual prior regions In order to expand from the tile level to the region level, the average features of the tiles in the same region are used as the region features. The specific feature map of the tile after dimensionality reduction is F p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region .

[0120] The present invention proposes an intra-class token balance module (ICTB), which integrates comprehensive reference information to replace the traditional nearest neighbor search method to optimize the retrieval process of memory data. Its goal is to extract the corresponding prototype feature map from each sample block-level feature map to be tested, and compare the relationship between the two to detect abnormal areas, while balancing the changes within the class through image-level class features.

[0121] See also Figure 3 The intra-class balancing module outputs the abnormal region location information map M corresponding to each image-level visual feature mem The implementation method includes the following steps:

[0122] The CTB module compares the image-level reference features corresponding to all reference samples with the image-level visual features F corresponding to each image-level visual feature. cls Perform class word balance to obtain the current image-level visual feature F cls The corresponding image-level weight

[0123]

[0124] The GPI module is used to map all tile-level reference feature maps corresponding to all reference samples and the current image-level visual feature F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Perform global prototype integration to obtain the prototype feature map F pro; Prototype feature map F pro And the spliced ​​block feature map F p ' atch Perform cosine similarity calculation to obtain the anomaly score of the i-th block in the anomaly map

[0125] According to the abnormal scores of all blocks, the abnormal area location information map M is obtained. mem .

[0126] In this preferred embodiment, the intra-class balancing module includes the following two sub-processes: Global Prototype Integration (GPI): extracting the tile-level prototype of the query feature; Class Token Balancing (CTB): using image-level features to balance the intra-class differences at the image level. And obtain the abnormal area location information map M mem The specific process integrates the global features of the reference samples and considers the confidence of the reference samples themselves and the differences between reference samples.

[0127] The present invention proposes a CTB module (Class-to-Term Balancing Module), which generates image-level weights by calculating the similarity between the class features of a test image and the class features in a memory library (i.e., the image-level reference features and tile-level reference feature maps corresponding to the reference sample image) to enhance the reference confidence across categories.

[0128] Specifically, we get the image-level weights The implementation methods include:

[0129] The image-level visual feature F corresponding to the current image-level visual feature cls The image-level features corresponding to the mth reference sample The cosine similarity of , and the reference weight of the mth reference sample is obtained

[0130] According to the reference weight Calculate the confidence weight of the mth reference sample

[0131] in, Indicates the image-level feature corresponding to the mth reference sample Respectively Calculate the minimum value of cosine similarity, is the image-level feature corresponding to the t-th reference sample The cosine similarity of , m and t are the indexes of the reference samples, and m≠t;

[0132] Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight W cls , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight W ref ;

[0133] According to the total reference weight W cls and the total confidence weight W ref , get the image-level weight in,

[0134]

[0135] ‖·‖2 is used to calculate the L2 norm.

[0136] In this preferred embodiment, the image-level weight obtained is It not only improves the differences in confidence between different references, but also balances the changes within the class.

[0137] In specific applications, the tile anomaly score is given One implementation method is as follows:

[0138] express and Calculate the cosine similarity. is the feature map of the spliced ​​block F p ' atch The features of the i-th block in , is the prototype feature map F pro The features of the i-th patch in .

[0139] The preferred embodiment given in this The specific formula expression can integrate the global normal mode of the reference sample into a block-level prototype feature map, and compare it with the block feature map of the sample to be tested to obtain a more global abnormal block score.

[0140] In specific applications, the abnormal area location information map M is given mem A specific implementation method is as follows:

[0141]

[0142] express With B patch After calculating the cosine similarity, take the largest value. is the feature map of the spliced ​​block F p ' atch The feature of the i-th block in B patchare all tile-level reference feature maps corresponding to all reference samples.

[0143] The M given in this preferred embodiment mem The specific formula expression of is used to consider the globality of the reference sample when scoring each tile, while retaining the local optimality of the tile-level feature map of the reference sample.

[0144] Furthermore, the GPI module is given to obtain the prototype feature map F pro The implementation methods include:

[0145] According to all the tile-level reference feature maps corresponding to all the reference samples and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Calculate the tile-level weight matrix W pro :

[0146]

[0147] <F p ' atch ,B patch > indicates F p ' atch With B patch Calculate the cosine similarity, B patch All tile-level reference feature maps corresponding to all reference samples;

[0148] For the tile-level weight matrix W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix

[0149]

[0150] according to and B patch , get the prototype feature map F pro , ‖·‖1 is used to calculate the L1 norm.

[0151] In this preferred method, in order to avoid irrelevant areas from being included in the prototype features, W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix And the prototype feature map F is given pro The specific implementation process of the prototype feature map is as follows. The prototype feature map extracted in this process retains the correlation between the block-level features of the reference sample and the sample to be tested, while also filtering out irrelevant areas, thereby improving the robustness and correctness of the prototype feature map.

[0152] See also Figure 4 ,The pre-trained VLPF visual encoder consists of an input layer, 24 self-attention encoding layers and an output layer that are cascaded in sequence;

[0153] And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer;

[0154] Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features F cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch ;

[0155] The visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer. patch After extracting the features of the abnormal area, it is sent to the next layer;

[0156] The image-level visual features F output by the 6th, 12th, 18th, and 24th self-attention encoding layers cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch , as the output of the pre-trained VLPF visual encoder.

[0157] In this embodiment, a pre-trained visual encoder is used, and a visual prior weighted attention module is added thereto, and the visual encoder is called a VLPF visual encoder. The self-attention encoding layer integrates the global semantic information of the image, makes up for the limitations of the local method, highlights important blocks through attention weights, suppresses irrelevant blocks, and can obtain diverse features. The self-attention encoding layer and the visual prior weighted attention module, as well as the selected 6th, 12th, 18th, and 24th self-attention encoding layer outputs as the outputs of the pre-trained VLPF visual encoder, can make the visual prior information better forward propagated in the VLPF visual encoder, and can better integrate the visual prior information into the tile-level feature map.

[0158] See also Figure 4 , a specific visual prior weighted attention module is used to focus on all tile-level visual feature maps F output by the previous layer patch The specific process of extracting abnormal area features is as follows:

[0159] All the tile-level visual feature maps F output by the previous layer patch After splicing the height and width of the image, we get the spliced ​​block feature map F p ' atchSend it to the second multiplier, and at the same time, according to the spliced ​​block feature map F p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0160] Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map Represents a spatial region with a length of HW and a channel number of 1;

[0161] The visual prior score map S vpk Binarize and get the visual prior mask M vpk ;

[0162] The visual prior mask M vpk Pixels above the threshold 0 are assigned weight w1, and pixels below or equal to the threshold 0 are assigned weight w2. The resulting weighted visual prior mask Sent to the second multiplier;

[0163] The second multiplier converts F p ' atch and After element-wise multiplication, a tile-level weighted feature map is generated And split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is HW and the number of channels is C patch space region, The height is H, the width is W, and the number of channels is C patch spatial domain.

[0164] In this preferred embodiment, a specific process of extracting features of abnormal regions using a visual prior weighted attention module is provided, which enables the block-level features to pay more attention to regions where abnormalities may exist, and ignore regions where abnormalities are unlikely to exist and background noise.

[0165] The present invention proposes Vision-Language Prior Fusion (VLPF). In order to evaluate the abnormal possibility of each block, the visual prior knowledge (VPK) score is calculated by the following formula, and the visual prior knowledge score of the block is specifically given. The specific expression is:

[0166]

[0167] express and The first m in the cosine similarity matrix k The highest value, and They are the feature maps of the spliced ​​blocks F p ' atch In the i-th and j-th tile-level features in the height dimension, the values ​​of i and j range from 1 to HW.

[0168] The preferred embodiment provides the following Specific expression, The larger the value of is, the higher the anomaly score corresponding to the abnormal block is, and the more obvious the anomaly is. This makes it easier to adjust the degree of anomaly that the model focuses on through a series of hyperparameters.

[0169] Principle analysis:

[0170] Industrial anomaly detection has become increasingly important in modern manufacturing processes. Traditional human-centric visual inspection methods are not only resource-intensive but also inefficient, which has driven the application of vision-language deep learning methods, such as the CLIP model. However, existing CLIP-based methods focus too much on the association between text and images, and ignore the inherent relationship within visual features.

[0171] The present invention proposes a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion, which is an innovative method - multimodal prior fusion CLIP (M 2 PF-CLIP). This method overcomes the key limitations of existing methods by combining visual-linguistic information and multi-scale fusion prior knowledge, thereby effectively improving the accuracy of anomaly detection. 2 PF-CLIP mainly consists of two core components: vision-language prior fusion and intra-class feature balance.

[0172] Vision-language prior fusion introduces region-level visual features into vision-language similarity calculation, which enhances the network's attention to potential abnormal regions. It achieves a higher level of feature understanding by using visual prior mask attention and combining it with connected region analysis to expand the mask from tile level to region level analysis.

[0173] Intra-class feature balancing reduces the interference of intra-class differences between multiple references by extracting global features from a single reference, aiming to utilize reference features more efficiently. It extracts corresponding prototype feature maps for each tile-level feature map of the sample to be tested, thereby improving the utilization efficiency of reference information. In addition, in order to balance the differences between different reference samples, intra-class feature balancing also comprehensively considers the changes in global features. The entire network combines visual principles and visual-text principles for industrial visual anomaly detection. It reduces intra-class changes in reference samples, integrates all features of image-level reference features and tile-level reference feature maps corresponding to all reference sample images in the memory library, and captures the global correlation of features, effectively solving the problem of insufficient consideration of image-level information.

[0174] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A few-sample industrial anomaly detection method based on multimodal prior knowledge fusion, characterized in that: The steps include: Training phase: Construct a training sample set, wherein each training sample in the training sample set includes a sample image to be tested, and a prompt word and a true value segmentation map corresponding to the sample image to be tested, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample image to be tested in the training sample set includes a normal sample image and an abnormal sample image, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word; The constructed visual-language prior fusion model is trained using each training sample. During the training process, the visual-language prior fusion model is used to output multiple image-level visual features F according to the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl , and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the visual-language prior fusion model are updated to complete the training; among them, the image-level visual feature F cls , Feature map of the spliced ​​blocks F p ' atch and language prior segmentation map M vl The number is the same, and there is a one-to-one correspondence; Reasoning phase: After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features F are output. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl ; After summing and averaging all language prior segmentation maps, the generated standard segmentation map M v ' l sent to the adder; K reference sample images are encoded by a pre-trained universal visual encoder to obtain multiple tile-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are sent to the constructed intra-class balancing module; K is an integer less than or equal to 4; Through the constructed intra-class balancing module, each image-level visual feature F cls And its corresponding spliced ​​block feature map F p ' atch The image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image are compared and analyzed respectively, and the abnormal area location information map M corresponding to each image-level visual feature is output. mem ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all the block-level visual feature maps corresponding to each image-level visual feature; After summing and averaging all abnormal area location information maps, the standard abnormal area location information map M′ is generated. mem Sent to the adder; the adder receives the standard segmentation map M v ' l and standard abnormal area location information map M′ mem After element-by-element addition, anomaly graph is generated to complete anomaly detection.

2. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 1 is characterized in that: The constructed visual-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module, and a cosine similarity calculation module; The pre-trained text encoder is used to encode the prompt word and obtain the text feature vector F t Send to the cosine similarity calculation module; The pre-trained VLPF visual encoder is used to output multiple image-level visual features F according to the sample image to be tested. cls , and multiple tile-level visual feature maps F corresponding to each image-level visual feature patch , and the image-level visual feature F cls As the output of the vision-language prior fusion model; The splicing module is used to combine multiple tile-level visual feature maps F corresponding to each image-level visual feature patch The height and width of the image are spliced ​​to obtain the spliced ​​block feature map F corresponding to each image-level visual feature. p ' atch , and the spliced ​​block feature map F p ' atch As the output of the vision-language prior fusion model; Multi-layer perceptron, used to map the spliced ​​tile feature map F corresponding to each image-level visual feature p ' atch Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map F corresponding to each image-level visual feature p ″ atch ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model; The visual prior information region generation module is used to generate the feature map F of each spliced ​​block. p ' atch Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature represents a spatial region with a length of N and a number of channels HW; The first multiplier is used to multiply the dimensionality-reduced block feature map F corresponding to each image-level visual feature. p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region ; Average pooling module, used to pool the region-level visual features F region After pooling, the pooled region-level visual feature F is obtained r ' egion ; The cosine similarity calculation module is used to calculate the text feature vector F corresponding to each image-level visual feature. t And the pooled region-level visual features F r ' egion Perform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature Represents a spatial region of length HW and channel number 1.

3. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 2 is characterized in that: Get the visual prior region R corresponding to each image-level visual feature vpk The implementation method includes the following steps: According to the feature map F of each spliced ​​block p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1; Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map is a spatial region with a length of HW and a channel number of 1; The visual prior score map S vpk Binarize and get the visual prior mask M vpk ; For the visual prior mask M vpk Perform connected domain analysis to obtain the visual prior region R vpk .

4. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 1 is characterized in that: The intra-class balancing module outputs the abnormal region location information map M corresponding to each image-level visual feature. mem The implementation method includes the following steps: The CTB module compares the image-level reference features corresponding to all reference samples with the image-level visual features F corresponding to each image-level visual feature. cls Perform class word balance to obtain the current image-level visual feature F cls The corresponding image-level weight The GPI module is used to map all tile-level reference feature maps corresponding to all reference samples and the current image-level visual feature F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Perform global prototype integration to obtain the prototype feature map F pro ; Prototype feature map F pro And the spliced ​​block feature map F p ' atch Perform cosine similarity calculation to obtain the anomaly score of the i-th block in the anomaly map According to the abnormal scores of all blocks, the abnormal area location information map M is obtained. mem .

5. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: The CTB module obtains image-level weights The implementation methods include: The image-level visual feature F corresponding to the current image-level visual feature cls The image-level features corresponding to the mth reference sample The cosine similarity of , and the reference weight of the mth reference sample is obtained According to the reference weight Calculate the confidence weight of the mth reference sample in, Indicates the image-level feature corresponding to the mth reference sample Respectively Calculate the minimum value of cosine similarity, is the image-level feature corresponding to the t-th reference sample The cosine similarity of , m and t are the indexes of the reference samples, and m≠t; Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight W cls , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight W ref ; According to the total reference weight W cls and the total confidence weight W ref , get the image-level weight in, ‖·‖2 is used to calculate the L2 norm.

6. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: express and Calculate the cosine similarity. is the feature map of the spliced ​​block F p ' atch The features of the i-th block in , is the prototype feature map F pro The features of the i-th block in ; express With B patch After calculating the cosine similarity, take the largest value, B patch are all tile-level reference feature maps corresponding to all reference samples.

7. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: The GPI module obtains the prototype feature map F pro The implementation methods include: According to all the tile-level reference feature maps corresponding to all the reference samples and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Calculate the tile-level weight matrix W pro : <F p ' atch ,B patch > indicates F p ' atch With B patch Calculate the cosine similarity, B patch All tile-level reference feature maps corresponding to all reference samples; For the tile-level weight matrix W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix according to and B patch , get the prototype feature map F pro , ‖·‖1 is used to calculate the L1 norm.

8. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 1 is characterized in that: The pre-trained VLPF visual encoder consists of an input layer, 24 self-attention encoding layers, and an output layer that are cascaded in sequence; And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer; Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features F cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch ; The visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer. patch After extracting the features of the abnormal area, it is sent to the next layer; The image-level visual features F output by the 6th, 12th, 18th, and 24th self-attention encoding layers cls and multiple block-level visual feature maps F corresponding to the image-level visual feature patch , as the output of the pre-trained VLPF visual encoder.

9. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 8 is characterized in that: The visual prior weighted attention module is used to integrate all the tile-level visual feature maps F output by the previous layer. patch The specific process of extracting abnormal area features is as follows: All the tile-level visual feature maps F output by the previous layer patch After splicing the height and width of the image, we get the spliced ​​block feature map F p ' atch Send it to the second multiplier, and at the same time, according to the spliced ​​block feature map F p ' atch , get the visual prior knowledge score of the i-th tile in the length dimension is a spatial region with a length of 1 and a channel number of 1; Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map Represents a spatial region with a length of HW and a channel number of 1; The visual prior score map S vpk Binarize and get the visual prior mask M vpk ; The visual prior mask M vpk Pixels above the threshold 0 are assigned weight w1, and pixels below or equal to the threshold 0 are assigned weight w2. The resulting weighted visual prior mask Sent to the second multiplier; The second multiplier converts F p ' atch and After element-wise multiplication, a tile-level weighted feature map is generated And split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is HW and the number of channels is C patch space region, The height is H, the width is W, and the number of channels is C patch spatial domain.

10. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 3 or 9, characterized in that: express and The first m in the cosine similarity matrix k The highest value, and They are the feature maps of the spliced ​​blocks F p ' atch In the i-th and j-th tile-level features in the height dimension, the values ​​of i and j range from 1 to HW.

Images