Few-shot industrial anomaly detection method based on multimodal prior knowledge fusion

Abstract

The few-sample industrial anomaly detection method based on multimodal prior knowledge fusion belongs to the field of industrial anomaly detection in computer vision. It solves the problem that traditional anomaly detection methods fail to fully consider the subtle differences between different samples and lack effective use of global information. The present invention introduces an innovative visual-language prior fusion method. The core of this method lies in two major innovations: constructing a visual-language prior fusion module and an intra-class balancing module. By effectively utilizing visual-language information and image-level reference features and tile-level reference feature maps corresponding to reference sample images, two core innovative components, visual-language prior fusion and intra-class feature balancing, are introduced to fuse visual prior and language prior information and balance changes within the class to achieve more accurate industrial anomaly detection. The present invention is mainly used in industrial anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of industrial anomaly detection in computer vision, and specifically relates to a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion. Background Art

[0002] In the industrial manufacturing process, industrial anomaly detection is a critical step in ensuring production quality. It not only directly impacts a company's cost control and market reputation, but also determines whether a product can maintain its competitive advantage. By detecting defects early in the production process, industrial anomaly detection can effectively prevent substandard products from entering the market, thereby reducing production losses. However, traditional industrial visual inspection relies on manual labor, which is time-consuming, labor-intensive, and inefficient, making it difficult to adapt to the demand for efficient inspection in modern industrial production.

[0003] In recent years, industrial anomaly detection has gradually become a research hotspot, with the goal of automatically identifying and locating anomalies in industrial product images through algorithms. However, existing methods face challenges such as a lack of negative samples, uneven sample distribution, or ambiguous defect definitions, making classifier construction more difficult. In particular, when there are only "normal" samples, the accuracy of demarcating category boundaries is limited. To address these issues, researchers initially adopted one-class classification (OCC) methods, but this approach, which requires training models separately for different objects, is difficult to meet the complex and changing needs of industry. In contrast, the introduction of multi-class classification and a unified framework has significantly improved detection efficiency. The core idea is to measure the difference between anomalies and normals through feature distance to achieve more robust anomaly detection.

[0004] With the development of pre-trained vision-language models (such as CLIP), industrial anomaly detection methods have been further expanded. These models, with their excellent feature representation capabilities, have particularly performed well in text-based object detection tasks, providing new insights into anomaly detection. Within the vision-language paradigm, detection methods identify abnormal regions by measuring the distance between visual features and text-based features. However, compared with the actual needs of industrial scenarios, such methods still have room for improvement. Language prior methods tend to identify the most prominent abnormal regions, for example, highlighting only the larger parts of capsule cracks while ignoring smaller cracks. In contrast, visual prior methods are more proactive and can identify the entire abnormal region, but may also include normal areas (such as labels). This complementarity opens up the possibility of fusing vision and language priors.

[0005] Despite this, current methods still face many limitations in scenarios with few samples. For example, when using multiple normal samples, the model is highly sensitive to background noise and position changes, and fails to fully account for subtle differences between samples. In addition, many methods treat reference samples equally and fail to assign differentiated weights to different samples, resulting in inaccurate detection results. At the same time, the nearest neighbor feature matching method based on the memory bank focuses more on local pixel features and lacks effective utilization of global information. These limitations indicate that how to efficiently integrate vision-vision and vision-language prior knowledge and mitigate the impact of intra-class variations while fusing memory bank features is an urgent problem that needs to be solved in current multimodal industrial anomaly detection research. Summary of the Invention

[0006] The purpose of this invention is to solve the problem that traditional anomaly detection methods fail to fully consider the subtle differences between different samples and lack effective use of global information. The present invention provides a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion.

[0007] The few-shot industrial anomaly detection method based on multimodal prior knowledge fusion includes the following steps:

[0008] Training phase:

[0009] Construct a training sample set, where each training sample in the training sample set includes a sample image to be tested, a prompt word corresponding to the sample image to be tested, and a true value segmentation map, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample images to be tested in the training sample set include normal sample images and abnormal sample images, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word;

[0010] The constructed vision-language prior fusion model is trained using each training sample. During the training process, the vision-language prior fusion model is used to output multiple image-level visual features F according to the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl , and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the vision-language prior fusion model are updated to complete the training; among them, the image-level visual feature F cls , Feature map of the spliced ​​block F p ' atch and language prior segmentation map M vl The number is the same and there is a one-to-one correspondence;

[0011] Reasoning stage:

[0012] After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features F are output. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl ;

[0013] After summing and averaging all language prior segmentation maps, the generated standard segmentation map M v ' l Sent to adder;

[0014] K reference sample images are encoded using a pre-trained universal visual encoder to obtain multiple tile-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are then sent to the constructed intra-class balancing module; K is an integer less than or equal to 4;

[0015] The constructed intra-class balancing module is used to transform each image-level visual feature F cls And its corresponding spliced ​​block feature map F p ' atch Perform feature comparison analysis with the image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image, and output the abnormal area location information map M corresponding to each image-level visual feature. mem ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all block-level visual feature maps corresponding to each image-level visual feature;

[0016] After summing and averaging all abnormal area location information maps, the standard abnormal area location information map M′ is generated. mem Sent to the adder; the adder receives the standard segmentation map M v ' l and standard abnormal area location information map M′ mem After element-by-element addition, anomaly graph is generated to complete anomaly detection.

[0017] Preferably, the constructed vision-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module and a cosine similarity calculation module;

[0018] The pre-trained text encoder is used to encode the prompt word features and obtain the text feature vector F t Send to the cosine similarity calculation module;

[0019] The pre-trained VLPF visual encoder is used to output multiple image-level visual features F according to the sample image to be tested. cls , and multiple tile-level visual feature maps F corresponding to each image-level visual feature patch , and the image-level visual features F cls As the output of the vision-language prior fusion model;

[0020] The splicing module is used to combine multiple tile-level visual feature maps F corresponding to each image-level visual feature patch The height and width of the image are spliced ​​together to obtain the spliced ​​block feature map F corresponding to each image-level visual feature. p ' atch , and the spliced ​​block feature map F p ' atch As the output of the vision-language prior fusion model;

[0021] Multi-layer perceptron, used to map the spliced ​​block feature map F corresponding to each image-level visual feature p ' atch Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map F corresponding to each image-level visual feature p ″ atch ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model;

[0022] The visual prior information region generation module is used to generate the feature map F of each spliced ​​block. p ' atch Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature represents a spatial region with a length of N and a number of channels HW;

[0023] The first multiplier is used to multiply the dimensionality-reduced block feature map F corresponding to each image-level visual feature. p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region ;

[0024] Average pooling module, used to pool the region-level visual features F region After pooling, the pooled regional visual feature F is obtained r ' egion ;

[0025] The cosine similarity calculation module is used to calculate the text feature vector F corresponding to each image-level visual feature t and the pooled region-level visual features F r ' egionPerform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature Represents a spatial region of length HW and channel number 1.

[0026] Preferably, the visual prior region R corresponding to each image-level visual feature is obtained. vpk The implementation includes the following steps:

[0027] According to the feature map F of each spliced ​​block p ' atch , get the visual prior knowledge score of the i-th block in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0028] Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map is a spatial region with a length of HW and a channel number of 1;

[0029] The visual prior score map S vpk Perform binarization to obtain the visual prior mask M vpk ;

[0030] For the visual prior mask M vpk Perform connected domain analysis to obtain the visual prior area R vpk .

[0031] Preferably, the intra-class balancing module outputs the abnormal region position information map M corresponding to each image-level visual feature. mem The implementation includes the following steps:

[0032] The CTB module is used to compare the image-level reference features corresponding to all reference samples with the image-level visual features F corresponding to each image-level visual feature. cls Perform class word balance to obtain the current image-level visual feature F cls The corresponding image-level weight

[0033]

[0034] Through the GPI module, all the reference samples corresponding to the block-level reference feature maps and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Perform global prototype integration to obtain the prototype feature map F pro ;

[0035] Prototype feature map F pro and the spliced ​​patch feature map F p' atch Perform cosine similarity calculation to obtain the anomaly score of the i-th block in the anomaly map

[0036] According to the abnormal scores of all blocks, the abnormal area location information map M is obtained mem .

[0037] Preferably, the CTB module obtains the image-level weights The implementation methods include:

[0038] The image-level visual feature F corresponding to the current image-level visual feature cls The image-level features corresponding to the mth reference sample The cosine similarity of , and the reference weight of the mth reference sample is obtained

[0039] According to the reference weight Calculate the confidence weight of the mth reference sample

[0040]

[0041] in, Indicates the image-level feature corresponding to the mth reference sample Respectively Calculate the minimum value of cosine similarity, is the image-level feature corresponding to the t-th reference sample The cosine similarity of , m and t are both indexes of reference samples, and m≠t;

[0042] Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight W cls , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight W ref ;

[0043] According to the total reference weight W cls and the total confidence weight W ref , get the image-level weight in,

[0044]

[0045] ‖·‖2 is used to calculate the L2 norm.

[0046] Preferably,

[0047] express and Calculate the cosine similarity. is the feature map of the spliced ​​block F p ' atch The features of the i-th block in , is the prototype feature map F pro The features of the i-th block in ;

[0048]

[0049] express With B patch After calculating the cosine similarity, take the largest value, B patch are all tile-level reference feature maps corresponding to all reference samples.

[0050] Preferably, the GPI module obtains the prototype feature map F pro The implementation methods include:

[0051] According to all the tile-level reference feature maps corresponding to all reference samples and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Calculate the tile-level weight matrix W pro :

[0052]

[0053] <F p ' atch ,B patch > indicates F p ' atch With B patch Perform cosine similarity calculation, B patch All tile-level reference feature maps corresponding to all reference samples;

[0054] For the tile-level weight matrix W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix

[0055]

[0056] according to and B patch , get the prototype feature map F pro , ‖·‖1 is used to calculate the L1 norm.

[0057] Preferably, the pre-trained VLPF visual encoder includes an input layer, 24 self-attention encoding layers and an output layer connected in cascade sequence;

[0058] And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer;

[0059] Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features F cls and multiple block-level visual feature maps F corresponding to the image-level visual features patch ;

[0060] The visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer patch After extracting the features of abnormal areas, they are sent to the next layer;

[0061] The image-level visual features F output by the 6th, 12th, 18th, and 24th self-attention encoding layers are cls and multiple block-level visual feature maps F corresponding to the image-level visual features patch , as the output of the pre-trained VLPF visual encoder.

[0062] Preferably, the visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer. patch The specific process of extracting abnormal area features is as follows:

[0063] All the tile-level visual feature maps F output by its previous layer patch After splicing the height and width dimensions, we get the spliced ​​block feature map F p ' atch Send it to the second multiplier, and at the same time, according to the spliced ​​block feature map F p ' atch , get the visual prior knowledge score of the i-th block in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0064] Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map Represents a spatial region with a length of HW and a channel number of 1;

[0065] The visual prior score map S vpk Perform binarization to obtain the visual prior mask M vpk ;

[0066] The visual prior mask M vpk Pixels above the threshold 0 are assigned weight w1, and pixels below or equal to the threshold 0 are assigned weight w2. The resulting weighted visual prior mask is Sent to the second multiplier;

[0067] The second multiplier converts F p ' atch and After element-wise multiplication, a tile-level weighted feature map is generated And split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is HW and the number of channels is C patch spatial region, The height is H, the width is W, and the number of channels is C patch spatial domain.

[0068] Preferably,

[0069] express and The first m in the cosine similarity matrix k The highest value, and They are the feature maps of the spliced ​​blocks F p ' atch In the height dimension, the i-th and j-th tile-level features, i and j both range from 1 to HW.

[0070] Advantages of the present invention:

[0071] The present invention provides a few-shot industrial anomaly detection method based on multimodal prior knowledge fusion. This method introduces an innovative vision-language prior fusion method. The core of this method lies in two major innovations: constructing a vision-language prior fusion module and an intra-class balancing module.

[0072] The constructed visual-language prior fusion module fuses the two modalities of vision and language, and utilizes language and visual information at the same time. Its design goal is to fuse visual prior information with language prior information, and to integrate visual prior information into block-level feature maps, so that it can pay more attention to areas where anomalies may exist and ignore those areas where anomalies are unlikely to exist, thereby enhancing the accuracy of anomaly detection.

[0073] The constructed intra-class balancing module integrates comprehensive reference information, replacing the traditional nearest neighbor search method to optimize the memory retrieval process. Its goal is to identify prototype patterns in query features. By extracting global features from a single reference, the intra-class balancing module reduces the interference of intra-class differences between multiple references, aiming to more efficiently utilize reference features. It extracts a corresponding prototype feature map for each block-level feature map of the sample to be tested and compares the relationship between the two to detect abnormal areas. At the same time, it balances intra-class variations through image-level class features, improving the efficiency of reference information utilization. In addition, to balance the differences between different reference samples, the intra-class feature balance also comprehensively considers the changes in global features.

[0074] Overall, the entire architecture combines visual principles and vision-text principles for industrial visual anomaly detection. It mitigates intra-class variations in reference samples and integrates all features of the data in the memory library (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images) to capture the global correlation of features. This effectively solves the problem of insufficient consideration of image-level information and provides new ideas and methods for vision-text detection in industrial anomaly detection.

[0075] The present invention effectively utilizes visual-linguistic information and data in the memory library (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images) to introduce two core innovative components: visual-linguistic prior fusion and intra-class feature balancing. This reduces the reliance on single information or traditional methods during the detection process, thereby improving the performance of the model in few-sample industrial anomaly detection and achieving more accurate industrial anomaly detection.

[0076] The present invention combines visual principles and vision-text principles for industrial visual anomaly detection. It reduces intra-class variations in reference samples and integrates all features of image-level reference features and tile-level reference feature maps corresponding to all reference sample images in the memory library to capture the global correlation of features, effectively solving the problem of insufficient consideration of image-level information. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a schematic diagram of the principle of the training phase of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion of the present invention;

[0078] Figure 2 This is a schematic diagram of the principle of the inference stage of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion of the present invention; wherein, is the image-level reference feature corresponding to the m-th reference sample, are all tile-level reference feature maps corresponding to the m-th reference sample;

[0079] Figure 3This is a schematic diagram of the internal principle of the intra-class balancing module;

[0080] Figure 4 This is a schematic diagram of the internal structure of the VLPF visual encoder. DETAILED DESCRIPTION

[0081] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0082] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other.

[0083] This paper aims to enhance the model's ability to process and analyze relevant data in industrial anomaly detection, improving detection accuracy and reliability. It proposes a method for few-shot industrial anomaly detection based on the fusion of multimodal prior knowledge. This method optimizes the feature extraction and fusion processes used in few-shot industrial anomaly detection by leveraging multimodal information and building innovative components.

[0084] Specific implementation method 1. Combination Figure 1 and Figure 2 As shown, the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion described in this embodiment includes the following steps:

[0085] Training phase:

[0086] Construct a training sample set, where each training sample in the training sample set includes a sample image to be tested, a prompt word corresponding to the sample image to be tested, and a true value segmentation map, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample images to be tested in the training sample set include normal sample images and abnormal sample images, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word;

[0087] The constructed vision-language prior fusion model is trained using each training sample. During the training process, the vision-language prior fusion model is used to output multiple image-level visual features F according to the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl, and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the vision-language prior fusion model are updated to complete the training; among them, the image-level visual feature F cls , Feature map of the spliced ​​block F p ' atch and language prior segmentation map M vl The number is the same and there is a one-to-one correspondence;

[0088] Reasoning stage:

[0089] After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features F are output. cls , multiple spliced ​​block feature maps F p ' atch and multiple language prior segmentation maps M vl ;

[0090] After summing and averaging all language prior segmentation maps, the generated standard segmentation map M v ' l Sent to adder;

[0091] K reference sample images are encoded using a pre-trained universal visual encoder to obtain multiple tile-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are then sent to the constructed intra-class balancing module; K is an integer less than or equal to 4;

[0092] The constructed intra-class balancing module is used to transform each image-level visual feature F cls And its corresponding spliced ​​block feature map F p ' atch Perform feature comparison analysis with the image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image, and output the abnormal area location information map M corresponding to each image-level visual feature. mem ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all block-level visual feature maps corresponding to each image-level visual feature;

[0093] After summing and averaging all abnormal area location information maps, the standard abnormal area location information map M′ is generated. mem Sent to adder;

[0094] The adder receives the standard partitioning diagram M v ' l and standard abnormal area location information map M′ mem After element-by-element addition, anomaly graph is generated to complete anomaly detection.

[0095] The core of the few-sample industrial anomaly detection method based on multimodal prior knowledge fusion described in this embodiment lies in two major innovations: constructing a vision-language prior fusion module and an intra-class balancing module.

[0096] The constructed visual-language prior fusion module fuses the two modalities of vision and language, and utilizes language and visual information at the same time. Its design goal is to fuse visual prior information with language prior information, and to integrate visual prior information into block-level feature maps, so that it can pay more attention to areas where anomalies may exist and ignore those areas where anomalies are unlikely to exist, thereby enhancing the accuracy of anomaly detection.

[0097] The constructed intra-class balancing module fully integrates the tile-level information of the reference samples into tile-level prototype features, preserving the global nature of the reference samples. Furthermore, the integration of tile-level prototype features is adjusted by taking into account the differences in image-level features between the test sample and the reference samples, as well as between the reference samples themselves, leading to more accurate anomaly localization.

[0098] The constructed intra-class balancing module integrates comprehensive reference information, replacing the traditional nearest neighbor search method to optimize the memory retrieval process. Its goal is to identify prototype patterns in query features. By extracting global features from a single reference, the intra-class balancing module reduces the interference of intra-class differences between multiple references, aiming to more efficiently utilize reference features. It extracts a corresponding prototype feature map for each block-level feature map of the sample to be tested and compares the relationship between the two to detect abnormal areas. At the same time, it balances intra-class variations through image-level class features, thereby improving the efficient use of reference information. Furthermore, to balance the differences between different reference samples, intra-class feature balancing also comprehensively considers changes in global features.

[0099] Overall, the entire architecture combines visual principles and vision-text principles for industrial visual anomaly detection. It mitigates intra-class variations in reference samples and integrates all features of the memory database data (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images) to capture the global correlation of features. This effectively solves the problem of insufficient consideration of image-level information and provides new ideas and methods for vision-text detection in industrial anomaly detection.

[0100] The present invention effectively utilizes visual-linguistic information and memory library data (i.e., image-level reference features and tile-level reference feature maps corresponding to reference sample images) to introduce two core innovative components: visual-linguistic prior fusion and intra-class feature balancing. This reduces the reliance on single information or traditional methods in the detection process, thereby improving the performance of the model in few-sample industrial anomaly detection and achieving more accurate industrial anomaly detection.

[0101] For details, see Figure 1,The constructed vision-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module and a cosine similarity calculation module;

[0102] The pre-trained text encoder is used to encode the prompt word features and obtain the text feature vector F t Send to the cosine similarity calculation module;

[0103] The pre-trained VLPF visual encoder is used to output multiple image-level visual features F according to the sample image to be tested. cls , and multiple tile-level visual feature maps F corresponding to each image-level visual feature patch , and the image-level visual features F cls As the output of the vision-language prior fusion model;

[0104] The splicing module is used to combine multiple tile-level visual feature maps F corresponding to each image-level visual feature patch The height and width of the image are spliced ​​together to obtain the spliced ​​block feature map F corresponding to each image-level visual feature. p ' atch , and the spliced ​​block feature map F p ' atch As the output of the vision-language prior fusion model;

[0105] Multi-layer perceptron, used to map the spliced ​​block feature map F corresponding to each image-level visual feature p ' atch Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map F corresponding to each image-level visual feature p ″ atch ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model;

[0106] The visual prior information region generation module is used to generate the feature map F of each spliced ​​block. p ' atch Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature represents a spatial region with a length of N and a number of channels HW;

[0107] The first multiplier is used to multiply the dimensionality-reduced block feature map F corresponding to each image-level visual feature. p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region ; Specifically,

[0108]

[0109] m α is a hyperparameter, R vpk The nth region in the length dimension, R vpk The eigenvalue of the nth region and the i-th channel in the length dimension;

[0110] Average pooling module, used to pool the region-level visual features F region After pooling, the pooled regional visual feature F is obtained r ' egion ;

[0111] The cosine similarity calculation module is used to calculate the text feature vector F corresponding to each image-level visual feature t and the pooled region-level visual features F r ' egion Perform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature Represents a spatial region of length HW and channel number 1.

[0112] In this preferred embodiment, the constructed visual-language prior fusion model introduces regional-level visual features into the visual-language similarity calculation, which enhances the network's attention to potential abnormal areas. By integrating visual prior knowledge, the visual information pays more attention to areas where abnormalities may exist and ignores areas where abnormalities are unlikely to exist.

[0113] In order to align with the text features, a multi-layer perceptron (MLP) is used to Perform dimensionality reduction and generate post-processing features

[0114] See also Figure 1 , further, we get the visual prior region R corresponding to each image-level visual feature vpk The implementation includes the following steps:

[0115] According to the feature map F of each spliced ​​block p ' atch , get the visual prior knowledge score of the i-th block in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0116] Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map is a spatial region with a length of HW and a channel number of 1;

[0117] The visual prior score map Svpk Perform binarization to obtain the visual prior mask M vpk ;

[0118] For the visual prior mask M vpk Perform connected domain analysis to obtain the visual prior area R vpk .

[0119] In this preferred embodiment, attention is generated by using a visual priori mask and combined with connected region analysis to expand the mask from the tile level to the region level analysis, thereby achieving a higher level of feature understanding. vpk Apply connected component analysis to generate binary visual prior regions In order to expand from the tile level to the region level, the average features of the tiles in the same region are used as the region features. The specific feature map of the tile after dimensionality reduction is F p ″ atch and the visual prior region R vpk Perform element-wise multiplication to obtain the region-level visual feature F region .

[0120] This paper proposes an intra-class token balance (ICTB) module, which optimizes the memory data retrieval process by integrating comprehensive reference information, replacing the traditional nearest neighbor search method. Its goal is to extract the corresponding prototype feature map from each sample tile-level feature map under test and compare the two to detect abnormal areas. It also balances intra-class variations through image-level class features.

[0121] See also Figure 3 The intra-class balancing module outputs the abnormal region location information map M corresponding to each image-level visual feature mem The implementation includes the following steps:

[0122] The CTB module is used to compare the image-level reference features corresponding to all reference samples with the image-level visual features F corresponding to each image-level visual feature. cls Perform class word balance to obtain the current image-level visual feature F cls The corresponding image-level weight

[0123]

[0124] Through the GPI module, all the reference samples corresponding to the block-level reference feature maps and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Perform global prototype integration to obtain the prototype feature map F pro; Prototype feature map F pro and the spliced ​​patch feature map F p ' atch Perform cosine similarity calculation to obtain the anomaly score of the i-th block in the anomaly map

[0125] According to the abnormal scores of all blocks, the abnormal area location information map M is obtained mem .

[0126] In this preferred embodiment, the intra-class balancing module includes the following two sub-processes: Global Prototype Integration (GPI): extracting tile-level prototypes of query features; Class Token Balancing (CTB): using image-level features to balance intra-class differences at the image level. And obtain the abnormal area location information map M mem The specific process integrates the global features of the reference samples and considers the confidence of the reference samples themselves and the differences between reference samples.

[0127] The present invention proposes a CTB module (Class-Term Balancing Module), which generates image-level weights by calculating the similarity between the class features of the test image and the class features in the memory library (i.e., the image-level reference features and tile-level reference feature maps corresponding to the reference sample images) to enhance the reference confidence across categories.

[0128] Specifically, we get the image-level weights The implementation methods include:

[0129] The image-level visual feature F corresponding to the current image-level visual feature cls The image-level features corresponding to the mth reference sample The cosine similarity of , and the reference weight of the mth reference sample is obtained

[0130] According to the reference weight Calculate the confidence weight of the mth reference sample

[0131] in, Indicates the image-level feature corresponding to the mth reference sample Respectively Calculate the minimum value of cosine similarity, is the image-level feature corresponding to the t-th reference sample The cosine similarity of , m and t are both indexes of reference samples, and m≠t;

[0132] Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight W cls , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight W ref ;

[0133] According to the total reference weight W cls and the total confidence weight W ref , get the image-level weight in,

[0134]

[0135] ‖·‖2 is used to calculate the L2 norm.

[0136] In this preferred embodiment, the image-level weight obtained It not only improves the confidence differences between different references, but also balances the changes within the class.

[0137] In specific applications, the tile anomaly score is given One implementation method is as follows:

[0138] express and Calculate the cosine similarity. is the feature map of the spliced ​​block F p ' atch The features of the i-th block in , is the prototype feature map F pro The features of the i-th patch in .

[0139] The preferred embodiment given in this The specific formula expression of can integrate the global normal pattern of the reference sample into a block-level prototype feature map, and compare it with the block feature map of the sample to be tested to obtain a more global abnormal block score.

[0140] In specific applications, the abnormal area location information map M is given mem A specific implementation method is as follows:

[0141]

[0142] express With B patch After calculating the cosine similarity, take the largest value. is the feature map of the spliced ​​block F p ' atch The feature of the i-th block in B patchare all tile-level reference feature maps corresponding to all reference samples.

[0143] The M given in this preferred embodiment mem The specific formula expression of is used to score each tile, which not only considers the globality of the reference sample but also retains the local optimality of the tile-level feature map of the reference sample.

[0144] Furthermore, the GPI module is given to obtain the prototype feature map F pro The implementation methods include:

[0145] According to all the tile-level reference feature maps corresponding to all reference samples and the current image-level visual features F cls The corresponding spliced ​​block feature map F p ' atch and image-level weights Calculate the tile-level weight matrix W pro :

[0146]

[0147] <F p ' atch ,B patch > indicates F p ' atch With B patch Perform cosine similarity calculation, B patch All tile-level reference feature maps corresponding to all reference samples;

[0148] For the tile-level weight matrix W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix

[0149]

[0150] according to and B patch , get the prototype feature map F pro , ‖·‖1 is used to calculate the L1 norm.

[0151] In this preferred method, in order to avoid irrelevant areas from being included in the prototype features, W pro Use the hyperparameter λ for sparse filtering to obtain a sparse tile-level weight matrix And the prototype feature map F is given pro The specific implementation process of the prototype feature map is that the prototype feature map extracted in this process retains the correlation between the block-level features of the reference sample and the sample to be tested, while also filtering out irrelevant areas, thereby improving the robustness and correctness of the prototype feature map.

[0152] See also Figure 4 ,The pre-trained VLPF visual encoder consists of an input layer, 24 self-attention encoding layers and an output layer that are cascaded in sequence;

[0153] And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer;

[0154] Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features F cls and multiple block-level visual feature maps F corresponding to the image-level visual features patch ;

[0155] The visual prior weighted attention module is used to add all the tile-level visual feature maps F output by the previous layer patch After extracting the features of abnormal areas, they are sent to the next layer;

[0156] The image-level visual features F output by the 6th, 12th, 18th, and 24th self-attention encoding layers are cls and multiple block-level visual feature maps F corresponding to the image-level visual features patch , as the output of the pre-trained VLPF visual encoder.

[0157] In this embodiment, a pre-trained visual encoder is used, and a visual prior weighted attention module is added to it. This visual encoder is called a VLPF visual encoder. The self-attention encoding layer integrates the global semantic information of the image, making up for the limitations of local methods, highlighting important blocks through attention weights, suppressing irrelevant blocks, and obtaining diverse features. The self-attention encoding layer and the visual prior weighted attention module, as well as the selected 6th, 12th, 18th, and 24th self-attention encoding layer outputs as the outputs of the pre-trained VLPF visual encoder, can better forward propagate the visual prior information in the VLPF visual encoder, and can better integrate the visual prior information into the tile-level feature map.

[0158] See also Figure 4 , a specific visual prior weighted attention module is used to focus on all the tile-level visual feature maps F output by its previous layer patch The specific process of extracting abnormal area features is as follows:

[0159] All the tile-level visual feature maps F output by its previous layer patch After splicing the height and width dimensions, we get the spliced ​​block feature map F p ' atchSend it to the second multiplier, and at the same time, according to the spliced ​​block feature map F p ' atch , get the visual prior knowledge score of the i-th block in the length dimension is a spatial region with a length of 1 and a channel number of 1;

[0160] Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map Represents a spatial region with a length of HW and a channel number of 1;

[0161] The visual prior score map S vpk Perform binarization to obtain the visual prior mask M vpk ;

[0162] The visual prior mask M vpk Pixels above the threshold 0 are assigned weight w1, and pixels below or equal to the threshold 0 are assigned weight w2. The resulting weighted visual prior mask is Sent to the second multiplier;

[0163] The second multiplier converts F p ' atch and After element-wise multiplication, a tile-level weighted feature map is generated And split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is HW and the number of channels is C patch spatial region, The height is H, the width is W, and the number of channels is C patch spatial domain.

[0164] In this preferred embodiment, a specific process of extracting features of abnormal regions using a visual prior weighted attention module is provided, which enables the block-level features to focus more on regions where abnormalities may exist, and ignore regions where abnormalities are unlikely to exist and background noise.

[0165] This paper proposes Vision-Language Prior Fusion (VLPF). In order to evaluate the abnormal possibility of each block, the visual prior knowledge (VPK) score is calculated by the following formula, and the visual prior knowledge score of the block is specifically given. The specific expression is:

[0166]

[0167] express and The first m in the cosine similarity matrix k The highest value, and They are the feature maps of the spliced ​​blocks F p ' atch In the height dimension, the i-th and j-th tile-level features, i and j both range from 1 to HW.

[0168] The preferred embodiment provides the following Specific expression, The larger the value of , the higher the anomaly score corresponding to the abnormal block, and the more obvious the abnormality is. This facilitates the subsequent adjustment of the degree of anomaly that the model focuses on through a series of hyperparameters.

[0169] Principle analysis:

[0170] Industrial anomaly detection is becoming increasingly important in modern manufacturing processes. Traditional, human-centric visual inspection methods are resource-intensive and inefficient, driving the application of vision-language deep learning methods, such as the CLIP model. However, existing CLIP-based methods overly focus on the association between text and images, neglecting the inherent relationships within visual features.

[0171] This paper proposes a few-sample industrial anomaly detection method based on multimodal prior knowledge fusion, which is an innovative method - multimodal prior fusion CLIP (M 2 PF-CLIP). This method overcomes the key limitations of existing methods by combining visual-linguistic information and prior knowledge of multi-scale fusion, thereby effectively improving the accuracy of anomaly detection. 2 PF-CLIP mainly consists of two core components: vision-language prior fusion and intra-class feature balance.

[0172] Vision-language prior fusion introduces region-level visual features into vision-language similarity calculations, enhancing the network's focus on potentially abnormal regions. By using visual prior mask attention and combining it with connected region analysis, it extends mask analysis from the tile level to the region level, achieving a higher level of feature understanding.

[0173] Intra-class feature balancing extracts global features from a single reference, reducing the interference of intra-class differences between multiple references. This approach aims to more efficiently utilize reference features. It extracts a corresponding prototype feature map for each tile-level feature map of the sample being tested, improving the utilization of reference information. Furthermore, to balance the differences between different reference samples, intra-class feature balancing also comprehensively considers global feature variations. The entire network combines visual and visual-text principles for industrial visual anomaly detection. It mitigates intra-class variation in reference samples, integrating image-level reference features and tile-level reference feature maps corresponding to all reference sample images in the memory library to capture global feature correlations, effectively addressing the issue of insufficient consideration of image-level information.

[0174] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A few-sample industrial anomaly detection method based on multimodal prior knowledge fusion, characterized by: The steps include: Training phase: Construct a training sample set, where each training sample in the training sample set includes a sample image to be tested, a prompt word corresponding to the sample image to be tested, and a true value segmentation map, and the sample image to be tested and the prompt word in each training sample are used as model input data, and the true value segmentation map in the training sample is used as model output data; wherein the sample images to be tested in the training sample set include normal sample images and abnormal sample images, and the prompt word corresponding to the normal sample image is used as a positive prompt word, and the prompt word corresponding to the abnormal sample image is used as a negative prompt word; The constructed visual-language prior fusion model is trained using each training sample. During the training process, the visual-language prior fusion model is used to output multiple image-level visual features based on the sample image to be tested in each training sample and the prompt word corresponding to the sample image to be tested. , multiple spliced ​​block feature maps and multiple language prior segmentation maps , and according to the deviation between each language prior segmentation map and the true value segmentation map corresponding to the sample image to be tested, the parameters of the vision-language prior fusion model are updated to complete the training; among them, the image-level visual features , Feature map of the spliced ​​blocks and language prior segmentation map The number is the same and there is a one-to-one correspondence; Reasoning stage: After the current sample image to be tested and its corresponding prompt word are input into the trained visual-language prior fusion model, multiple image-level visual features are output. , multiple spliced ​​block feature maps and multiple language prior segmentation maps ; After summing and averaging all language prior segmentation maps, the standard segmentation map generated Sent to adder; Through pre-trained universal visual encoder The reference sample images are encoded to obtain multiple block-level reference feature maps and one image-level reference feature corresponding to each reference sample image, which are sent to the constructed intra-class balancing module; is an integer less than or equal to 4; The constructed intra-class balancing module combines the image-level visual features And the corresponding spliced ​​block feature map Perform feature comparison analysis with the image-level reference features and multiple block-level reference feature maps corresponding to each reference sample image, and output the abnormal area location information map corresponding to each image-level visual feature. ; Wherein, the number of the multiple block-level reference feature maps corresponding to each reference sample image is the same as the number of all block-level visual feature maps corresponding to each image-level visual feature; After summing and averaging all abnormal area location information maps, the standard abnormal area location information map is generated. Sent to the adder; the adder receives the standard segmentation diagram and standard abnormal area location information map After element-by-element addition, anomaly graph is generated to complete anomaly detection.

2. The method for detecting anomalies in industrial environments using a small number of samples based on multimodal prior knowledge fusion according to claim 1, characterized in that: The constructed vision-language prior fusion model includes a pre-trained text encoder, a pre-trained VLPF visual encoder, a splicing module, a visual prior information region generation module, a multi-layer perceptron, a first multiplier, an average pooling module, and a cosine similarity calculation module; Pre-trained text encoder, used to encode the prompt word features and obtain the text feature vector Send to the cosine similarity calculation module; Pre-trained VLPF visual encoder, used to output multiple image-level visual features based on the sample image to be tested , and multiple tile-level visual feature maps corresponding to each image-level visual feature , and image-level visual features As the output of the vision-language prior fusion model; The splicing module is used to combine multiple tile-level visual feature maps corresponding to each image-level visual feature The height and width of the image are spliced ​​together to obtain the spliced ​​block feature map corresponding to each image-level visual feature. , and the tile feature map after splicing As the output of the vision-language prior fusion model; Multi-layer perceptron, used to map the spliced ​​image tiles corresponding to each image-level visual feature Perform dimensionality reduction processing to obtain the dimensionality-reduced block feature map corresponding to each image-level visual feature ; And the parameters of the multilayer perceptron are used as the parameters of the vision-language prior fusion model; Visual prior information region generation module is used to generate the feature map of each spliced ​​block Extract the prior information area to obtain the visual prior area corresponding to each image-level visual feature , Indicates the length is , the number of channels is spatial region; The first multiplier is used to multiply the dimensionality-reduced block feature map corresponding to each image-level visual feature. and visual prior regions Perform element-wise multiplication to obtain regional-level visual features ; Average pooling module for region-level visual features After pooling, the pooled regional visual features are obtained ; Cosine similarity calculation module, used to calculate the text feature vector corresponding to each image-level visual feature and pooled region-level visual features Perform cosine similarity calculation to obtain the language prior segmentation map corresponding to each image-level visual feature , Indicates the length is , the spatial region with a channel number of 1.

3. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 2 is characterized in that: Get the visual prior area corresponding to each image-level visual feature The implementation includes the following steps: According to the feature map of each spliced ​​block , get the length dimension Visual prior knowledge score of each patch , is a spatial region with a length of 1 and a channel number of 1; Concatenate the visual prior knowledge scores of all tiles in the height dimension to obtain the visual prior score map , The length is , the spatial region with a channel number of 1; Visual prior score map Perform binarization to obtain the visual prior mask ; Visual prior mask Perform connected domain analysis to obtain visual prior regions .

4. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 1 is characterized in that: The intra-class balancing module outputs the abnormal area location information map corresponding to each image-level visual feature The implementation includes the following steps: Through the CTB module, the image-level reference features corresponding to all reference samples are compared with the image-level visual features corresponding to each image-level visual feature. Perform class word balance to obtain the current image-level visual features The corresponding image-level weight ; Through the GPI module, all tile-level reference feature maps corresponding to all reference samples and the current image-level visual features The corresponding spliced ​​block feature map and image-level weights Perform global prototype integration to obtain the prototype feature map ; Prototype feature map and the feature map of the spliced ​​blocks Perform cosine similarity calculation to obtain the first Tile anomaly score ; According to the abnormal scores of all tiles, the abnormal area location information map is obtained .

5. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: The CTB module obtains image-level weights The implementation methods include: The image-level visual features corresponding to the current image-level visual features With the The image-level features corresponding to the reference samples The cosine similarity of Reference weight of reference samples ; According to the reference weight , calculate the The confidence weight of the reference samples ; ； in, Indicates taking the The image-level features corresponding to the reference samples Respectively Calculate the minimum value of cosine similarity, For the The image-level features corresponding to the reference samples The cosine similarity of are the indexes of the reference samples, and ; Concatenate the reference weights of all reference samples along the length dimension to obtain the total reference weight , concatenate the confidence weights of all reference samples along the length dimension to obtain the total confidence weight ; According to the total reference weight and the total confidence weight , get the image-level weight ;in, To calculate the L2 norm.

6. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: ； express and Calculate the cosine similarity. Feature map of the spliced ​​block Middle The characteristics of the block, Prototype feature map Middle The characteristics of each tile; express and After calculating the cosine similarity, take the largest value. are all tile-level reference feature maps corresponding to all reference samples.

7. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 4 is characterized in that: GPI module obtains prototype feature map The implementation methods include: According to all the tile-level reference feature maps corresponding to all reference samples and the current image-level visual features The corresponding spliced ​​block feature map and image-level weights , calculate the tile-level weight matrix : ； express and Calculate the cosine similarity. All tile-level reference feature maps corresponding to all reference samples; For the tile-level weight matrix Using Hyperparameters Perform sparse filtering to obtain a sparse tile-level weight matrix : ； according to and , get the prototype feature map , , To calculate the L1 norm.

8. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 2 is characterized in that: The pre-trained VLPF visual encoder consists of an input layer, 24 self-attention encoding layers, and an output layer connected in cascade sequence; And four visual prior weighted attention modules are respectively set between the 6th and 7th self-attention encoding layers, between the 12th and 13th self-attention encoding layers, between the 18th and 19th self-attention encoding layers, and between the 24th self-attention encoding layer and the output layer; Each self-attention encoding layer is used to integrate the global semantic information of the image and output image-level visual features and multiple tile-level visual feature maps corresponding to the image-level visual feature ; Visual prior weighted attention module for all tile-level visual feature maps output by its previous layer After extracting the features of abnormal areas, they are sent to the next layer; The image-level visual features output by the 6th, 12th, 18th, and 24th self-attention encoding layers and multiple tile-level visual feature maps corresponding to the image-level visual feature , as the output of the pre-trained VLPF visual encoder.

9. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 8 is characterized in that: The visual prior weighted attention module is used to add all tile-level visual feature maps output by its previous layer. The specific process of extracting abnormal area features is as follows: All tile-level visual feature maps output by its previous layer After splicing the height and width dimensions, the spliced ​​block feature map is obtained Send it to the second multiplier, and at the same time, according to the feature map of the spliced ​​block , get the length dimension Visual prior knowledge score of each patch , is a spatial region with a length of 1 and a channel number of 1; Concatenate the visual prior knowledge scores of all tiles in the length dimension to obtain the visual prior score map , Indicates the length is , the spatial region with a channel number of 1; Visual prior score map Perform binarization to obtain the visual prior mask ; The visual prior mask Pixels above the threshold 0 are given weight , pixels below or equal to the threshold 0 are given weight , the obtained weighted visual prior mask Sent to the second multiplier; The second multiplier will and After element-wise multiplication, a tile-level weighted feature map is generated , and split its length dimension into two dimensions to obtain the tile-level weighted feature map after dimension splitting As the input of the next layer; The length is , the number of channels is The spatial region, For height , width is , the number of channels is spatial domain.

10. The method for industrial anomaly detection based on a small number of samples and multimodal prior knowledge fusion according to claim 3 or 9, characterized in that: express and The cosine similarity matrix between The highest value, and They are the feature maps of the spliced ​​blocks respectively In the height dimension and tile-level features, and The value range is from 1 to .

Images