A multi-scale visual-textual collaborative anomaly detection method

Abstract

The application discloses a kind of multiscale vision-text collaborative anomaly detection methods, it is related to computer vision and natural language processing technical field.The method utilizes labeled sample to establish temporary normal, abnormal memory bank, and jointly trains to basic vision-text scoring model and learnable retrieval contrast fusion scoring network;According to initial score result, dynamically construct target domain normal, abnormal memory bank, obtain query feature using multiscale self-adaptive fusion module, carry out prototype search to double memory bank before k Most relevant prototype retrieval, relationship coding, attention aggregation, normal reconstruction and abnormal enhancement, contrast fusion generates memory branch abnormal heat map and image level score, and then with the semantic branch generated by text encoder, obtain final pixel level abnormal heat map and image level abnormal score.The method can dynamically adjust memory bank according to target domain distribution and strengthen vision-text collaboration, which is beneficial to improve the robustness and positioning accuracy of anomaly detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, and in particular to a multi-scale visual-text collaborative anomaly detection method. Background Technology

[0002] Anomaly detection methods based on vision-language pre-trained models have been widely applied in industrial quality inspection, medical image analysis, surface defect identification, and intelligent monitoring. These methods typically utilize image encoders to extract visual features, input textual prompts such as "no defects" or "defective," and then use the cosine similarity between visual and textual features to achieve image-level discrimination and pixel-level localization. Typical zero-shot anomaly detection methods include WinCLIP, AnomalyCLIP, and AF-CLIP based on the CLIP model, all of which have achieved good detection results.

[0003] Regarding this direction, there are several existing approaches: one type of method is based on diffusion models and text embedding for anomaly detection. It uses a graph-to-text generation model to extract text information from image or video frames, then obtains semantic features through a text encoder, and reconstructs image and text features using a diffusion model to complete anomaly identification. Another type of method is based on graph neural networks and language feature guidance for anomaly detection. It segments the image into multiple image blocks and constructs a graph structure. It aggregates local and global features hierarchically through graph neural networks, stores normal sample features in a feature memory, and uses the minimum distance between the test features and the memory as the anomaly criterion.

[0004] However, the above methods still have significant shortcomings: First, multi-scale feature fusion mostly adopts fixed weights or simple splicing strategies, making it difficult to dynamically adjust the contribution of each scale according to different samples, resulting in unstable responses to anomalies with large scale differences, such as small cracks, edge damage, and large-area stains; Second, visual and text alignment mostly stays at the feature level for direct similarity calculation, with insufficient cross-modal interaction depth and weak anomaly semantic expression; Third, many methods only use normal features as a one-sided reference or only build a single memory bank, lacking explicit modeling of the distribution of anomaly prototypes. When normal textures and anomaly textures are highly similar or the anomaly morphology is complex and varied, the model is prone to false detection and false negative detection, and the overall discrimination boundary is not clear enough; Fourth, memory-based anomaly detection methods mostly rely on fixed reference features pre-built during the training phase, making it difficult to adaptively update according to the texture distribution and anomaly patterns of the current scene to be detected. When the distribution shifts in the target domain, the fixed memory bank's ability to represent the current anomaly pattern is insufficient, resulting in decreased detection robustness. Summary of the Invention

[0005] The purpose of this invention is to propose a multi-scale visual-text collaborative anomaly detection method to solve the problems of existing anomaly detection methods, such as difficulty in adapting to target domain distribution shifts, unstable multi-scale responses, insufficient cross-modal interaction, and lack of explicit modeling of anomaly prototypes, thereby improving the localization and discrimination accuracy of anomaly detection.

[0006] To achieve the above objectives, this invention proposes a multi-scale visual-text collaborative anomaly detection method, the specific steps of which are as follows: Step S1: Obtain training image samples labeled with normal or abnormal categories, and construct a basic visual-text model including a visual encoder, a text encoder, and a learnable retrieval contrast fusion scoring network. Use the labeled training samples to establish temporary normal memory and temporary abnormal memory. Step S2: Jointly train the basic visual-text scoring model and the learnable retrieval comparison fusion scoring network based on the temporary normal memory bank and the temporary abnormal memory bank; Step S3: Input the set of images to be detected into the trained basic visual-text scoring model to obtain the initial normal or abnormal confidence of each image to be detected; Step S4: Divide the set of images to be detected into suspected normal images and suspected abnormal images according to the initial normal or abnormal confidence level, and dynamically construct the target domain normal memory and the target domain abnormal memory from the suspected normal images and suspected abnormal images respectively. Step S5: Extract visual features of the current image to be detected through a visual encoder, and use a multi-scale adaptive fusion module to adaptively weight and fuse the multi-scale features extracted in parallel under different scale windows to obtain the query features. Step S6: Compare the query features with the prototypes in the target domain normal memory and the target domain abnormal memory, respectively. k The most relevant prototypes are retrieved to obtain a set of candidate prototypes; Step S7: Based on the candidate prototype set, the query features are subjected to relational encoding, attention aggregation, normal reconstruction or anomaly enhancement and contrast fusion processing using a learnable retrieval contrast fusion scoring network to generate a memory branch pixel-level anomaly heatmap and a memory branch image-level anomaly score. Step S8: Use a text encoder to map normal and abnormal prompts into semantic vectors, and generate a semantic branch pixel-level anomaly heatmap and a semantic branch image-level anomaly score. Step S9: The pixel-level anomaly heatmap of the memory branch and the pixel-level anomaly heatmap of the semantic branch are weighted and fused to obtain the final pixel-level anomaly heatmap. The image-level anomaly score of the memory branch and the image-level anomaly score of the semantic branch are weighted and fused to obtain the final image-level anomaly score.

[0007] Preferably, in step S2, the total loss function for jointly training the basic visual-text scoring model and the learnable retrieval contrast fusion scoring network based on the temporary normal memory bank and the temporary abnormal memory bank is as follows: ; in, This is the total loss function value. For pixel-level segmentation loss, For image-level classification loss, For visual-text alignment loss, , , These are the weighting coefficients; Pixel-level segmentation loss The calculation formula is as follows: ; in, For the focus of loss, Loss due to dice, This is the balance coefficient; Image-level classification loss The calculation formula is as follows: ; in, For sample labels, =0 indicates a normal sample. =1 indicates an abnormal sample. To predict the probability that an image is an anomaly for the model. It is the natural logarithm function; Visual-text alignment loss The calculation formula is as follows: ; in, For image features, Embedded as normal text. Embedding abnormal text, For text embedding corresponding to the true category of the current sample, For similarity function, For temperature coefficient, It is a natural exponential function.

[0008] Preferably, in step S4, dynamically constructing a target domain normal memory and a target domain abnormal memory from suspected normal images and suspected abnormal images includes: selecting low-abnormal response patches from suspected normal images as normal candidate patches, selecting high-abnormal response patches from suspected abnormal images as abnormal candidate patches, extracting features from normal candidate patches and features from abnormal candidate patches, and performing similarity redundancy removal processing and clustering compression processing on the features of normal candidate patches and features of abnormal candidate patches, respectively, to form a prototype set of the target domain normal memory and a prototype set of the target domain abnormal memory.

[0009] Preferably, in step S5, the multi-scale adaptive fusion module is used to adaptively weight and fuse the multi-scale features extracted in parallel under different scale windows, as follows: Step S51: Perform parallel feature extraction on visual features through multiple scale windows of different sizes to obtain multi-scale visual representations corresponding to each scale, as shown in the following formula: ; in, For multi-scale visual representation, For scale s The corresponding feature extraction function, As a visual feature, This refers to the size of the scale window; Step S52: Process the multi-scale visual representation through learnable weight generation branches and normalize it using softmax to obtain the adaptive weights corresponding to each scale, as shown in the following formula: ; in, For scale s Adaptive weights, For scale s The corresponding learnable weight generation function, For scale r The corresponding learnable weight generation function; Step S53: Dynamically weight and fuse the multi-scale visual representations according to adaptive weights, and normalize the fusion result to obtain the query features, as shown in the following formula: ; in, To query features, For normalization operations, This is element-wise multiplication.

[0010] Preferably, in step S6, the query features are compared with the prototypes in the target domain normal memory and the target domain abnormal memory, respectively. k The most relevant prototypes were retrieved, as follows: Step S61: Calculate the cosine similarity between the query feature and each prototype feature in the normal memory of the target domain, and each prototype feature in the abnormal memory of the target domain. The formula is as follows: ; in, For the first i Each query plot feature These are prototype features in the memory bank. Is it a normal branch or an abnormal branch? It is a 2-norm. i To query the tile index, i =1,2,…,P, where P is the number of query tiles contained in the image to be detected; Step S62: Select the top two with the highest similarity respectively. k A normal prototype and an abnormal prototype: ; in, To query the map The corresponding prototype retrieval set; when hour, To query the map The corresponding normal prototype retrieval set; when hour, To query the map The corresponding abnormal prototype retrieval set, The preset number of search prototypes; The retrieved number Each prototype is denoted as .

[0011] Preferably, in step S7, a learnable retrieval contrast fusion scoring network is used to encode the query features into relationships, as follows: For querying the map and the prototype obtained from the search The query features, prototype features, differences between the two, and element-wise interaction features are concatenated to obtain the relation input. : ; The relation encoding result is: ; in, Indicates feature splicing, This represents element-wise multiplication. This represents a relational coding network consisting of fully connected layers, normalization layers, and nonlinear activation layers. To query the map With prototype The relational encoding result.

[0012] Preferably, in step S7, an attention aggregation of query features is performed using a learnable retrieval contrast fusion scoring network, as follows: Based on the relation encoding results, the attention weights corresponding to each retrieval prototype are obtained through a learnable attention weight generation function, as shown in the following formula: ; in, To retrieve the attention weights corresponding to the prototype, For learnable attention weight generation functions, The summation index for attention weight normalization; The normal prototypes in the normal prototype retrieval set are weighted and aggregated according to attention weights to obtain the normal reference representation: ; The anomaly prototypes in the anomaly prototype retrieval set are weighted and aggregated according to attention weights to obtain the anomaly reference representation: ; in, For normal reference representation, The attention weights for the normal branch. For the retrieved first j A normal prototype This is an abnormal reference representation. The attention weights for the abnormal branches, For the retrieved first j An abnormal prototype.

[0013] Preferably, in step S7, a learnable retrieval contrast fusion scoring network is used to reconstruct the query features normally or enhance them abnormally, as follows: By utilizing a learnable normal reconstruction network, normal pattern reconstruction is performed on the query features based on normal reference representations to obtain normal reconstructed features: ; Based on the query features and the normal reconstruction features, the normal deviation features are obtained: ; in, This is a normal reconstruction characteristic. For a learnable and normal network reconstruction, To determine the degree to which query features deviate from the normal pattern; Generate anomaly-enhanced gating weights based on query features and anomaly reference representations: ; The query features are augmented according to the anomaly augmentation gating weights to obtain the anomaly augmented features: ; The corresponding anomaly enhancement response is: ; in, To enhance the gating weights for abnormal situations, For anomaly-enhanced gating generation functions, It is the Sigmoid activation function. This is an abnormal enhancement feature. Enhance the response to anomalies.

[0014] Preferably, in step S7, a learnable retrieval contrast fusion scoring network is used to perform contrast fusion processing on the query features and generate a memory branch pixel-level anomaly heatmap and a memory branch image-level anomaly score, as follows: The normal deviation features, abnormal enhancement features, and difference features between the normal reference representation and the abnormal reference representation are concatenated to obtain the contrast fusion input: ; The contrast fusion features are input into a multilayer perceptron for contrast fusion to obtain the memory branch fusion features: ; The memory branch fusion features corresponding to all image patches are used to restore the spatial feature map. The spatial feature map is then decoded using a convolutional decoding head to obtain the pixel-level anomaly heatmap of the memory branch: ; Pooling is performed on the pixel-level anomaly heatmap of the memory branch to obtain the image-level anomaly score of the memory branch: ; in, To compare fusion features, For memory branch fusion features, To compare and fuse multilayer perceptrons, For convolutional decoding head, For feature recovery operations, To memorize pixel-level anomaly heatmaps for branching. For pooling operations, The memory branch image-level anomaly score.

[0015] Preferably, in step S9, the pixel-level anomaly heatmap of the memory branch and the pixel-level anomaly heatmap of the semantic branch are weighted and fused together, and the image-level anomaly score of the memory branch and the image-level anomaly score of the semantic branch are weighted and fused together, as follows: The memory branch output and the semantic branch output are weighted according to the fusion weight coefficient and the semantic branch adjustment coefficient to obtain the final pixel-level anomaly heatmap and the final image-level anomaly score: ; ; in, , To integrate the weighting coefficients, This is the semantic branch adjustment coefficient. To memorize pixel-level anomaly heatmaps for branching. For semantic branches, pixel-level anomaly heatmaps For memory branch image-level anomaly scores, Image-level anomaly scores are assigned to semantic branches.

[0016] Therefore, this invention proposes a multi-scale visual-text collaborative anomaly detection method, the beneficial effects of which are as follows: (1) The present invention automatically adjusts the contribution ratio of local details and global context according to the sample content through the multi-scale adaptive fusion module, which can improve the model’s ability to perceive anomalies of different sizes and different spatial diffusion ranges, and is conducive to stabilizing the detection response of anomalies of different scales.

[0017] (2) This invention uses learnable relational encoding, attention aggregation, normal reconstruction and abnormal enhancement and contrast fusion to deeply model the relationship between query features and normal and abnormal prototypes. It can explicitly characterize the two-sided distribution of normal and abnormal patterns, which is conducive to clearly dividing the normal and abnormal discrimination boundary and reducing false detection and false negative detection when normal texture and abnormal texture are highly similar or abnormal morphology is complex and varied.

[0018] (3) This invention can enhance the cross-modal collaboration between vision and text by jointly judging the semantic branch and memory branch at the text end, enrich the semantic expression of abnormality, and help improve the accuracy of image-level discrimination and pixel-level positioning.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multi-scale visual-text collaborative anomaly detection method according to the present invention; Figure 2 This is a schematic diagram of the multi-scale adaptive fusion module in this invention; Figure 3 This is a schematic diagram of the dual-memory retrieval and comparison fusion network in this invention. Detailed Implementation

[0021] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 like Figure 1 As shown, this invention provides a multi-scale visual-text collaborative anomaly detection method. The overall process includes six stages: training a basic visual-text scoring model, coarse screening of the target domain in the detection stage, construction of a dual memory database, learnable retrieval and comparative fusion, text-based collaborative discrimination, and final fusion output. In the training stage, the basic visual-text scoring model and the learnable scoring network are jointly optimized; in the detection stage, a dual memory database is dynamically constructed based on the initial scoring results, and fine detection is performed.

[0024] During the training phase, a basic visual-text scoring model is constructed, comprising a visual encoder, a text encoder, and a learnable retrieval contrastive fusion scoring network. Temporary normality and anomaly memory banks are established using training image samples labeled with normal or anomaly categories, used to train the learnable retrieval contrastive fusion scoring network. To ensure that both the basic visual-text scoring model and the learnable retrieval contrastive fusion scoring network simultaneously possess pixel-level anomaly localization capabilities, image-level normal / anomaly discrimination capabilities, and visual-text semantic alignment capabilities, they are jointly trained.

[0025] In the detection phase, the set of images to be detected is first input into a basic visual-text scoring model to obtain an initial normal / abnormal confidence score for each image, and based on this, the images are roughly divided into suspected normal images and suspected anomalous images. Subsequently, low-abnormal response patches are selected from the suspected normal images as normal candidate patches, and after redundancy removal and clustering compression, a normal memory library for the target domain is formed; high-abnormal response patches are selected from the suspected anomalous images as anomalous candidate patches, and after redundancy removal and clustering compression, an anomalous memory library for the target domain is formed.

[0026] When performing fine-grained detection on a single image, the image is first input into a visual encoder, and enhanced query features are obtained by combining the image with a multi-scale adaptive fusion module. Then, the query features are compared with prototypes in both the target domain's normal memory and the target domain's abnormal memory. kThe most relevant prototypes are retrieved to obtain a candidate prototype set. Then, a memory branch representation is generated through learnable relation encoding, attention aggregation, normal reconstruction / abnormal enhancement, and contrastive fusion using a multilayer perceptron. The memory branch pixel-level anomaly heatmap and memory branch image-level anomaly score are then output via a convolutional decoder. Simultaneously, semantic embeddings are generated on the text side based on normal and abnormal prompts, and the semantic branch pixel-level anomaly heatmap and semantic branch image-level anomaly score are calculated. Finally, a weighted fusion strategy is used to obtain the final pixel-level anomaly heatmap and the final image-level anomaly score.

[0027] like Figure 2 As shown, the multi-scale adaptive fusion module extracts input features in parallel within windows of different scales (e.g., 1×1, 3×3, 5×5) to obtain the corresponding multi-scale visual representation: ; in, For multi-scale visual representation, For scale s The corresponding feature extraction function, As a visual feature, The scale window size is defined; subsequently, a learnable weight generation branch is introduced, which obtains adaptive weights for each scale through softmax normalization. ; in, For scale s Adaptive weights, For scale s The corresponding learnable weight generation function, For scale r The corresponding learnable weight generation function; Dynamically weighted fusion of multi-scale features is performed, and the fusion result is normalized to obtain the query feature set, as shown in the following formula: ; in, To query the feature set, For normalization operations, This is an element-wise multiplication. This module can automatically adjust the contribution ratio of local details and global context based on the input sample content, which helps improve the model's ability to perceive anomalies of different sizes and different spatial diffusion ranges.

[0028] Suppose the image to be detected contains P query patches, and its query feature set, target domain normal memory, and target domain abnormal memory are represented as follows: , ; in, Indicates the first Each query plot feature and These represent the first and second parts of the normal memory bank and the abnormal memory bank, respectively. One prototype feature.

[0029] like Figure 3 As shown, for any query tile feature The system calculates the cosine similarity between the feature and the prototype features in the normal and abnormal memory banks, respectively. : ; in, These are prototype features in the memory bank. Is it a normal branch or an abnormal branch? It is a 2-norm. i To query the tile index, i =1,2,…,P, where P is the number of query tiles contained in the image to be detected; the system selects the top [number] tiles with the highest similarity. k A normal prototype and an abnormal prototype: ; in, To query the map The corresponding prototype retrieval set; when hour, To query the map The corresponding normal prototype retrieval set; when hour, To query the map The corresponding abnormal prototype retrieval set, The preset number of prototypes to be searched; the retrieved prototypes will be... Each prototype is denoted as .

[0030] For querying the map and the prototype obtained from the retrieval The query features, prototype features, differences between the two, and element-wise interaction features are concatenated to obtain the relation input. : ; The relation encoding result is: ; in, Indicates feature splicing, This represents element-wise multiplication. This represents a relational coding network consisting of fully connected layers, normalization layers, and nonlinear activation layers. To query the map With prototype The relational encoding result.

[0031] Based on the relation encoding results, the attention weights corresponding to each retrieval prototype are obtained through a learnable attention weight generation function, as shown in the following formula: ; in, To retrieve the attention weights corresponding to the prototype, For learnable attention weight generation functions, The summation index for attention weight normalization; The normal prototypes in the normal prototype retrieval set are weighted and aggregated according to attention weights to obtain the normal reference representation: ; The anomaly prototypes in the anomaly prototype retrieval set are weighted and aggregated according to attention weights to obtain the anomaly reference representation: ; in, For normal reference representation, The attention weights for the normal branch. For the retrieved first j A normal prototype This is an abnormal reference representation. The attention weights for the abnormal branches, For the retrieved first j An abnormal prototype.

[0032] The normal reconstruction branch utilizes a learnable normal reconstruction network to reconstruct the query features using normal patterns based on normal reference representations, resulting in normally reconstructed features: ; Based on the query features and the normal reconstruction features, the normal deviation features are obtained: ; in, This is a normal reconstruction characteristic. For a learnable and normal network reconstruction, To determine the degree to which query features deviate from the normal pattern; normal tiles can be reconstructed well from normal prototypes, therefore Smaller; abnormal patches deviate from the normal feature distribution, therefore Relatively large.

[0033] The anomaly enhancement branch first generates anomaly enhancement gating weights based on query features and anomaly reference representations: ; The query features are augmented according to the anomaly augmentation gating weights to obtain the anomaly augmented features: ; The corresponding anomaly enhancement response is: ; in, To enhance the gating weights for abnormal situations, For anomaly-enhanced gating generation functions, It is the Sigmoid activation function. This is an abnormal enhancement feature. Enhance the response to anomalies.

[0034] The normal deviation features, abnormal enhancement features, and difference features between the normal reference representation and the abnormal reference representation are concatenated to obtain the contrast fusion input: ; The contrast fusion features are input into a multilayer perceptron for contrast fusion to obtain the memory branch fusion features: ; The memory branch fusion features corresponding to all image patches are used to restore the spatial feature map. The spatial feature map is then decoded using a convolutional decoding head to obtain the pixel-level anomaly heatmap of the memory branch: ; Pooling is performed on the pixel-level anomaly heatmap of the memory branch to obtain the image-level anomaly score of the memory branch: ; in, To compare fusion features, For memory branch fusion features, To compare and fuse multilayer perceptrons, For convolutional decoding head, For feature recovery operations, To memorize pixel-level anomaly heatmaps for branching. For memory branch image-level anomaly scores, For pooling operations, max pooling, average pooling, average pooling of the highest response region, or learnable pooling can be used.

[0035] Since the relation encoding, attention aggregation, normal reconstruction, anomaly enhancement and contrast fusion modules all contain learnable parameters and their outputs participate in pixel-level and image-level joint supervision, the learnable scoring network can continuously optimize the relationship modeling ability between query features and normal and abnormal prototypes during training.

[0036] In text-based collaborative discrimination, the text encoder first maps "normal prompts" and "abnormal prompts" into semantic vectors, and generates pixel-level anomaly heatmaps and image-level anomaly scores for the semantic branches. Finally, the system uses a weighted fusion strategy to obtain the final anomaly output: ; ; in, , To integrate the weighting coefficients, This is the semantic branch adjustment coefficient. To memorize pixel-level anomaly heatmaps for branching. For semantic branches, pixel-level anomaly heatmaps For memory branch image-level anomaly scores, Image-level anomaly scores are assigned to semantic branches.

[0037] In this embodiment, to enable the basic visual-text scoring model and the learnable retrieval comparison fusion scoring network to simultaneously possess pixel-level anomaly localization capability, image-level normal / anomaly discrimination capability, and visual-text semantic alignment capability, the two are jointly trained, and the total loss function is as follows: ; in, This is the total loss function value. For pixel-level segmentation loss, For image-level classification loss, For visual-text alignment loss, , , These are the weighting coefficients; Pixel-level segmentation loss The calculation formula is as follows: ; in, For the focus of loss, Loss due to dice, This is the balance coefficient; Image-level classification loss The calculation formula is as follows: ; in, For sample labels, =0 indicates a normal sample. =1 indicates an abnormal sample. To predict the probability that an image is an anomaly for the model. It is the natural logarithm function; Visual-text alignment loss The calculation formula is as follows: ; in, For image features, Embedded as normal text. Embedding abnormal text, For text embedding corresponding to the true category of the current sample, For similarity function, For temperature coefficient, It is a natural exponential function.

[0038] The method in this embodiment can be applied to zero-sample anomaly detection scenarios such as industrial quality inspection, medical image analysis, surface defect identification, and intelligent monitoring. It outputs image-level anomaly discrimination results and pixel-level anomaly scoring maps to the image to be detected, thereby achieving anomaly identification and precise positioning of the target surface.

[0039] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0040] Therefore, this invention provides a multi-scale visual-text collaborative anomaly detection method. By using multi-scale adaptive fusion, bilateral prototype explicit modeling, target domain dynamic memory, and text-memory joint discrimination, it solves the problems of distribution shift, unstable multi-scale response, and insufficient cross-modal interaction, thereby improving the robustness and localization accuracy of anomaly detection.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-scale visual-text collaborative anomaly detection method, characterized in that, Includes the following steps: Step S1: Obtain training image samples labeled with normal or abnormal categories, and construct a basic visual-text model including a visual encoder, a text encoder, and a learnable retrieval contrast fusion scoring network. Use the labeled training samples to establish temporary normal memory and temporary abnormal memory. Step S2: Jointly train the basic visual-text scoring model and the learnable retrieval comparison fusion scoring network based on the temporary normal memory bank and the temporary abnormal memory bank; Step S3: Input the set of images to be detected into the trained basic visual-text scoring model to obtain the initial normal or abnormal confidence of each image to be detected; Step S4: Divide the set of images to be detected into suspected normal images and suspected abnormal images according to the initial normal or abnormal confidence level, and dynamically construct the target domain normal memory and the target domain abnormal memory from the suspected normal images and suspected abnormal images respectively. Step S5: Extract visual features of the current image to be detected through a visual encoder, and use a multi-scale adaptive fusion module to adaptively weight and fuse the multi-scale features extracted in parallel under different scale windows to obtain the query features. Step S6: Compare the query features with the prototypes in the target domain normal memory and the target domain abnormal memory, respectively. k The most relevant prototypes are retrieved to obtain a set of candidate prototypes; Step S7: Based on the candidate prototype set, the query features are subjected to relational encoding, attention aggregation, normal reconstruction or anomaly enhancement and contrast fusion processing using a learnable retrieval contrast fusion scoring network to generate a memory branch pixel-level anomaly heatmap and a memory branch image-level anomaly score. Step S8: Use a text encoder to map normal and abnormal prompts into semantic vectors, and generate a semantic branch pixel-level anomaly heatmap and a semantic branch image-level anomaly score. Step S9: The pixel-level anomaly heatmap of the memory branch and the pixel-level anomaly heatmap of the semantic branch are weighted and fused to obtain the final pixel-level anomaly heatmap. The image-level anomaly score of the memory branch and the image-level anomaly score of the semantic branch are weighted and fused to obtain the final image-level anomaly score.

2. The multi-scale visual-text collaborative anomaly detection method according to claim 1, characterized in that, In step S2, the total loss function for jointly training the basic visual-text scoring model and the learnable retrieval contrast fusion scoring network based on the temporary normal memory bank and the temporary abnormal memory bank is as follows: ； in, This is the total loss function value. For pixel-level segmentation loss, For image-level classification loss, For visual-text alignment loss, , , These are the weighting coefficients; Pixel-level segmentation loss The calculation formula is as follows: ； in, For the focus of loss, Loss due to dice, This is the balance coefficient; Image-level classification loss The calculation formula is as follows: ； in, For sample labels, =0 indicates a normal sample. =1 indicates an abnormal sample. To predict the probability that an image is an anomaly for the model. It is the natural logarithm function; Visual-text alignment loss The calculation formula is as follows: ； in, For image features, Embedded as normal text. Embedding abnormal text, For text embedding corresponding to the true category of the current sample, For similarity function, For temperature coefficient, It is a natural exponential function.

3. The multi-scale visual-text collaborative anomaly detection method according to claim 2, characterized in that, In step S4, a normal memory and an abnormal memory of the target domain are dynamically constructed from suspected normal images and suspected abnormal images. This includes: selecting low-abnormal response patches from suspected normal images as normal candidate patches, selecting high-abnormal response patches from suspected abnormal images as abnormal candidate patches, extracting features from normal candidate patches and features from abnormal candidate patches, and performing similarity deredundancy processing and clustering compression processing on the features of normal candidate patches and features from abnormal candidate patches, respectively, to form a prototype set of the normal memory of the target domain and a prototype set of the abnormal memory of the target domain.

4. The multi-scale visual-text collaborative anomaly detection method according to claim 3, characterized in that, In step S5, the multi-scale adaptive fusion module is used to adaptively weight and fuse the multi-scale features extracted in parallel under different scale windows, as follows: Step S51: Perform parallel feature extraction on visual features through multiple scale windows of different sizes to obtain multi-scale visual representations corresponding to each scale, as shown in the following formula: ； in, For multi-scale visual representation, For scale s The corresponding feature extraction function, As a visual feature, This refers to the size of the scale window; Step S52: Process the multi-scale visual representation through learnable weight generation branches and normalize it using softmax to obtain the adaptive weights corresponding to each scale, as shown in the following formula: ； in, For scale s Adaptive weights, For scale s The corresponding learnable weight generation function, For scale r The corresponding learnable weight generation function; Step S53: Dynamically weight and fuse the multi-scale visual representations according to adaptive weights, and normalize the fusion result to obtain the query features, as shown in the following formula: ； in, To query features, For normalization operations, This is element-wise multiplication.

5. The multi-scale visual-text collaborative anomaly detection method according to claim 4, characterized in that, In step S6, the query features are compared with the prototypes in the target domain normal memory and the target domain abnormal memory, respectively. k The most relevant prototypes were retrieved, as follows: Step S61: Calculate the cosine similarity between the query feature and each prototype feature in the normal memory of the target domain, and each prototype feature in the abnormal memory of the target domain. The formula is as follows: ； in, For the first i Each query plot feature These are prototype features in the memory bank. Is it a normal branch or an abnormal branch? It is a 2-norm. i To query the tile index, i =1,2,…,P, where P is the number of query tiles contained in the image to be detected; Step S62: Select the top two with the highest similarity respectively. k A normal prototype and an abnormal prototype: ； in, To query the map The corresponding prototype retrieval set; when hour, To query the map The corresponding normal prototype retrieval set; when hour, To query the map The corresponding abnormal prototype retrieval set, The preset number of search prototypes; The retrieved number Each prototype is denoted as .

6. The multi-scale visual-text collaborative anomaly detection method according to claim 5, characterized in that, In step S7, a learnable retrieval contrast fusion scoring network is used to encode the query features into relationships, as follows: For querying the map and the prototype obtained from the search The query features, prototype features, differences between the two, and element-wise interaction features are concatenated to obtain the relation input. : ； The relation encoding result is: ； in, Indicates feature splicing, This represents element-wise multiplication. This represents a relational coding network consisting of fully connected layers, normalization layers, and nonlinear activation layers. To query the map With prototype The relational encoding result.

7. The multi-scale visual-text collaborative anomaly detection method according to claim 6, characterized in that, In step S7, an attention aggregation of query features is performed using a learnable retrieval contrast fusion scoring network, as detailed below: Based on the relation encoding results, the attention weights corresponding to each retrieval prototype are obtained through a learnable attention weight generation function, as shown in the following formula: ； in, To retrieve the attention weights corresponding to the prototype, For learnable attention weight generation functions, The summation index for attention weight normalization; The normal prototypes in the normal prototype retrieval set are weighted and aggregated according to attention weights to obtain the normal reference representation: ； The anomaly prototypes in the anomaly prototype retrieval set are weighted and aggregated according to attention weights to obtain the anomaly reference representation: ； in, For normal reference representation, The attention weights for the normal branch. For the retrieved first j A normal prototype This is an abnormal reference representation. The attention weights for the abnormal branches, For the retrieved first j An abnormal prototype.

8. The multi-scale visual-text collaborative anomaly detection method according to claim 7, characterized in that, In step S7, a learnable retrieval contrast fusion scoring network is used to reconstruct the query features normally or enhance them abnormally, as follows: By utilizing a learnable normal reconstruction network, normal pattern reconstruction is performed on the query features based on normal reference representations to obtain normal reconstructed features: ； Based on the query features and the normal reconstruction features, the normal deviation features are obtained: ； in, This is a normal reconstruction characteristic. For a learnable and normal network reconstruction, To determine the degree to which query features deviate from the normal pattern; Generate anomaly-enhanced gating weights based on query features and anomaly reference representations: ； The query features are augmented according to the anomaly augmentation gating weights to obtain the anomaly augmented features: ； The corresponding anomaly enhancement response is: ； in, To enhance the gating weights for abnormal situations, For anomaly-enhanced gating generation functions, It is the Sigmoid activation function. This is an abnormal enhancement feature. Enhance the response to anomalies.

9. The multi-scale visual-text collaborative anomaly detection method according to claim 8, characterized in that, In step S7, a learnable retrieval contrast fusion scoring network is used to perform contrast fusion processing on the query features and generate a memory branch pixel-level anomaly heatmap and a memory branch image-level anomaly score, as detailed below: The normal deviation features, abnormal enhancement features, and difference features between the normal reference representation and the abnormal reference representation are concatenated to obtain the contrast fusion input: ； The contrast fusion features are input into a multilayer perceptron for contrast fusion to obtain the memory branch fusion features: ； The memory branch fusion features corresponding to all image patches are used to restore the spatial feature map. The spatial feature map is then decoded using a convolutional decoding head to obtain the pixel-level anomaly heatmap of the memory branch: ； Pooling is performed on the pixel-level anomaly heatmap of the memory branch to obtain the image-level anomaly score of the memory branch: ； in, To compare fusion features, For memory branch fusion features, To compare and fuse multilayer perceptrons, For convolutional decoding head, For feature recovery operations, To memorize pixel-level anomaly heatmaps for branching. For pooling operations, The memory branch image-level anomaly score.

10. The multi-scale visual-text collaborative anomaly detection method according to claim 9, characterized in that, In step S9, the pixel-level anomaly heatmap of the memory branch and the pixel-level anomaly heatmap of the semantic branch are weighted and fused together, and the image-level anomaly score of the memory branch and the image-level anomaly score of the semantic branch are weighted and fused together, as follows: The memory branch output and the semantic branch output are weighted according to the fusion weight coefficient and the semantic branch adjustment coefficient to obtain the final pixel-level anomaly heatmap and the final image-level anomaly score: ； ； in, , To integrate the weighting coefficients, This is the semantic branch adjustment coefficient. To memorize pixel-level anomaly heatmaps for branching. For semantic branches, pixel-level anomaly heatmaps For memory branch image-level anomaly scores, Image-level anomaly scores are assigned to semantic branches.

Images