An unsupervised continuous anomaly detection method and system for multimodal cue memory

Abstract

The application discloses a kind of multi-modal prompt memory's unsupervised continuous anomaly detection method and system, it is related to computer vision and industrial image detection technical field.The method includes the following steps: obtaining industrial image data set;Unsupervised anomaly detection model is built, including visual branch and text branch, respectively for extracting visual features and text features, in unsupervised anomaly detection model learnable visual prompt and learnable text prompt are introduced, then the adaptive fusion mechanism is used to fuse visual features and text features, and the anomaly detection result is obtained;Learnable visual prompt and learnable text prompt in unsupervised anomaly detection model are trained using data set;Industrial data are detected using trained unsupervised anomaly detection model.The application can fuse multi-modal information in learning process and have continuous learning ability, realizes the efficient unsupervised anomaly detection of industrial product.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and industrial image detection technology, and in particular to a multi-modal prompt memory unsupervised continuous anomaly detection method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] The early anomaly detection method of industrial products mainly relies on supervised learning, that is, a classification model is constructed by training a large number of labeled normal samples and abnormal samples. However, it is difficult to obtain large-scale labeled data in actual industrial production scenarios. Therefore, as a learning paradigm that does not depend on labeled data, unsupervised anomaly detection has emerged and received extensive attention. Unsupervised anomaly detection learns the feature distribution of normal data and determines the data deviating from the normal mode as abnormal, thereby avoiding the dependence on a large amount of labeled data.

[0004] However, the existing unsupervised anomaly detection method has many limitations, such as the one-to-one anomaly detection paradigm with increasing computational burden as the number of classes increases, and the unified model prone to catastrophic forgetting when the product changes frequently, which limits its effectiveness in practical applications. Most mainstream methods are based on single-modal data features, such as using only visual information for anomaly detection. However, in complex industrial scenarios, single-modal features often fail to fully capture the diversity and complexity of the normal mode. For example, when detecting some surface defects, relying only on visual features may not be able to accurately distinguish between changes in normal texture and real defects, resulting in a high false positive rate. Multi-modal data (such as text description, depth information, etc.) usually contains more rich information, which can describe objects from different angles and help to more accurately depict the normal mode and identify anomalies. However, multi-modal data means more information, which is more susceptible to noise in unsupervised learning due to the lack of label guidance, thereby reducing sensitivity. In addition, the data splicing of multi-modal data in the unsupervised learning process will exacerbate the curse of dimensionality, affecting the accuracy and efficiency of the detection results. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application aims to provide a multi-modal prompt memory unsupervised continuous anomaly detection method and system, which can integrate multi-modal information during the learning process and have continuous learning ability, realizing efficient unsupervised anomaly detection of industrial products.

[0006] In order to achieve the above-mentioned purpose, the present application is realized by the following technical scheme:

[0007] The first aspect of the present application provides a multi-modal prompt memory unsupervised continuous anomaly detection method, comprising the following steps:

[0008] acquire an industrial image to constitute a data set, and pre-process the industrial image;

[0009] construct an unsupervised anomaly detection model, including a visual branch and a text branch, respectively used for extracting visual features and text features, introduce a learnable visual hint and a learnable text hint in the unsupervised anomaly detection model, use the visual hint to guide the extraction of the visual features, use the text hint to guide the extraction of the text features, and then use an adaptive fusion mechanism to fuse the visual features and the text features to obtain an anomaly detection result;

[0010] use the data set to train the learnable visual hint and the learnable text hint in the unsupervised anomaly detection model;

[0011] use the trained unsupervised anomaly detection model to perform anomaly detection on industrial data.

[0012] The second aspect of the present application provides an unsupervised continuous anomaly detection system with multi-modal hint memory, comprising:

[0013] a data acquisition module configured to acquire an industrial image to constitute a data set, and pre-process the industrial image;

[0014] a model construction module configured to construct an unsupervised anomaly detection model, including a visual branch and a text branch, respectively used for extracting visual features and text features, introduce a learnable visual hint and a learnable text hint in the unsupervised anomaly detection model, use the visual hint to guide the extraction of the visual features, use the text hint to guide the extraction of the text features, and then use an adaptive fusion mechanism to fuse the visual features and the text features to obtain an anomaly detection result;

[0015] a model training module configured to use the data set to train the learnable visual hint and the learnable text hint in the unsupervised anomaly detection model;

[0016] an anomaly detection module configured to use the trained unsupervised anomaly detection model to perform anomaly detection on industrial data.

[0017] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is adapted to be loaded and executed by a processor to perform the steps in the multi-modal hint memory unsupervised continuous anomaly detection method according to the first aspect of the present application.

[0018] The fourth aspect of the present application provides a computer device, which comprises:

[0019] a processor adapted to execute a computer program;

[0020] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the multi-modal prompt memory unsupervised continuous anomaly detection method.

[0021] The above one or more technical solutions have the following beneficial effects:

[0022] The application discloses a kind of multi-modal prompt memory unsupervised continuous anomaly detection method and system, first to industrial image data is preprocessed, and it is divided into training set and test set.After that, the pseudo-anomaly image generation and refinement feature prompt module of anomaly detection model, text prompt design and learnable text prompt module, continuous multi-modal prompt memory library and the self-adaptive fusion mechanism of anomaly detection model defect semantic guidance are built.And use training set to train overall framework, use test set to infer.In the aspect of multi-modal learning, try to combine various modal data such as text, image, improve the performance of anomaly detection by cross-modal feature alignment and fusion.In continuous learning, by introducing continuous learning framework, structure contrast learning and other ways to alleviate the problem of catastrophic forgetting.Therefore, the application effectively avoids the dependence on a large number of labeled data, and the problem of catastrophic forgetting in multiple scene industries, and provides an unsupervised continuous industrial image anomaly detection framework model.

[0023] In view of the existing multi-modal data processing process

[0024] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 For the multi-modal prompt memory unsupervised continuous anomaly detection method flow chart in embodiment one of the present application;

[0027] Figure 2 For the unsupervised anomaly detection model framework structure diagram in embodiment one of the present application;

[0028] Figure 3 For the learnable text prompt module structure diagram in embodiment one of the present application;

[0029] Figure 4A structure diagram of a multi-modal prompt memory bank for continuously prompting in the embodiment one of the present application;

[0030] Figure 5 A structure diagram of a self-adaptive fusion mechanism guided by defect semantics in the embodiment one of the present application. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0032] It should be noted that the terms used herein are only intended to describe specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component and / or combination thereof;

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0034] Embodiment one:

[0035] The embodiment one of the present application provides an unsupervised continuous anomaly detection method for multi-modal prompt memory, as shown in Figure 1 and Figure 2 , comprising the following steps:

[0036] S1: Obtain an industrial image constituting data set, and pre-process the industrial image.

[0037] In a specific embodiment, the pre-processed data set is divided into a training set and a test set, the unsupervised anomaly detection model is trained by using the training set, and the unsupervised anomaly detection model is tested by using the test set, wherein the training set only contains normal images of the industrial image, and is set for the scenario of unsupervised anomaly detection. The industrial image categories in the test set are consistent with the industrial image categories in the training set, and the test set contains various abnormal images and normal images of the industrial image.

[0038] First, these industrial images are pre-processed. The industrial images are converted by using the following formula:

[0039] .

[0040] wherein, denotes the converted dataset, is to represent the uniform size of all industrial images, denotes the image dimension, is the scaled size, representing the height and width of the image. denotes the size of the region cropped from the center of the image, wherein is the target size. and denote the mean and variance. denotes the linear mapping of the numerical range of the image from [0, 255] to [0.0, 1.0], becoming a tensor that can be input into the neural network. The training set is denoted as , and the test set is denoted as contains normal images and abnormal images, denotes normal images, denotes abnormal images.

[0041] S2: Constructing an unsupervised anomaly detection model, including a visual branch and a text branch, respectively used for extracting visual features and text features, and introducing a learnable visual hint and a learnable text hint in the unsupervised anomaly detection model.

[0042] S21: Preliminary construction of an unsupervised anomaly detection model.

[0043] S211: Generating abnormal images by adding noise using a pseudo abnormal image data generation module.

[0044] In a specific embodiment, in order to increase the diversity of the training set, make the model contact more different types of abnormal situations, learn more extensive feature representations, and thus improve its generalization ability in unseen abnormal situations. A pseudo abnormal image data generation module is constructed in the unsupervised anomaly detection model, and for each channel of the image , the following operations are performed:

[0045] ,

[0046] .

[0047] wherein, denotes the image processed by the pseudo abnormal image data generation module, is a set of real numbers, is the pixel coordinate of the image, denotes the Gaussian noise, and . denotes the noise mask. If , which means that the position is added with noise; which means that no noise is added.

[0048] S212: Constructing a visual branch and a text branch, and extracting visual features and text features by using the visual branch and the text branch respectively.

[0049] In a specific embodiment, the unsupervised anomaly detection model adopts a double-branch collaborative inference method to extract visual features and text features, and then combines a dynamic normalization strategy with an adaptive fusion mechanism to effectively fuse visual and text features. The double-branch includes a visual branch and a text branch, and both the visual branch and the text branch are composed of a pre-trained ViT model. The input of the visual branch and the input of the text branch both include normal image data, image data after adding noise, and text data, wherein the normal image data comes from the training set, the image data after adding noise comes from the pseudo-anomaly image data generation module, and the text data is generated by a text semantic template.

[0050] In the visual branch, a pre-trained visual transformer is used as a visual backbone network to extract multi-level features, and the 5th layer features are used for task identity recognition and anomaly detection due to their balanced semantic and detailed information. Specifically, first, a task identifier is constructed by farthest point sampling to calculate the nearest neighbor distance and realize fast matching of the task identity to obtain normal features ; then, the extracted visual features and the normal features stored in the multi-modal prompt memory bank are calculated by block-wise nearest neighbor distance to generate preliminary image block-level anomaly scores :

[0051] .

[0052] wherein, is the normal feature of the multi-modal prompt memory bank, is the corresponding category.

[0053] In the text branch, the 5th layer features are extracted due to their rich local detailed characteristics, and are calculated by cross-modal cosine similarity with the embedding vector of the learnable text prompt to obtain text-guided anomaly scores :

[0054] .

[0055] wherein, is the feature local weight, i denotes the spatial position index in the image, j denotes the word or token index in the text prompt, is the dot product of vectors, is the Euclidean norm of vectors.

[0056] S22: Introducing learnable visual cues and learnable text cues in the unsupervised anomaly detection model.

[0057] wherein the visual cues are used to guide the extraction of visual features, the text cues are used to guide the extraction of text features, and then an adaptive fusion mechanism is used to fuse the visual features and the text features to obtain an anomaly detection result.

[0058] S221: Constructing a refined feature cue module to introduce hierarchical refined learnable visual cues to guide unsupervised anomaly detection of visual features.

[0059] In a specific embodiment, a hierarchical refined visual cue is introduced to guide unsupervised anomaly detection. Task-aware visual cues are composed of learnable vectors with dimension lxC, where, denotes the length of the cue, and C denotes the channel dimension. These visual cues are initialized by uniform distribution (Uniform (0, 1)):

[0060] .

[0061] In the task adaptation and inference stage, the visual cues are integrated into the input features of each layer of the pre-trained visual branch ViT hierarchical model to perform prefix tuning. This process realizes the layer-by-layer propagation of task-specific information, which is formally represented as:

[0062] .

[0063] wherein, denotes the output feature from the layer of the backbone network, denotes the feature from layer, denotes the visual cue learned specifically for the layer to facilitate fine-grained task adaptation, and t denotes task t.

[0064] S222: Introducing learnable text cues to guide unsupervised anomaly detection of text features.

[0065] S2221: Constructing a text cue design module to introduce a text semantic template containing learnable text cues. ​

[0066] In one specific embodiment, as shown in Figure 3 a learnable text semantic template S is introduced, which is in the form of:

[0067] S = "A photo of a [class] with [P]".

[0068] where [class] represents a specific class label (such as bottle), and [P] is a text prompt composed of N learnable vectors, denoted as:

[0069] .

[0070] Set N = 5, and all vectors are initialized to standard normal distribution, which is used to dynamically model the semantic feature distribution of normal samples. The text feature extraction module, i.e., the backbone used to extract text features in the text branch, mainly comes from the pre-trained text encoder of CLIP. Denote the output of the text encoder as :

[0071] .

[0072] where is the learnable text semantic prompt template, is the pre-trained text encoder.

[0073] S2222: Construct a learnable text prompt module to optimize learning of the learnable text prompt by fusing image inherent semantic information.

[0074] In one specific embodiment, to solve the problem that the artificially designed text prompt template has limitations in describing normal or abnormal information of industrial images, and the actual image semantic information is not considered in the text prompt, therefore, the embodiment optimizes learning of the learnable text prompt by fusing image inherent semantic information, thereby realizing higher efficiency of abnormal segmentation and detection. Specifically, in the task adaptive stage, the learnable text prompt is optimized by using noise-enhanced pseudo abnormal samples for contrastive learning . The process is as follows:

[0075] First, the images of the training set in step S211 are subjected to image feature capture by a pre-trained visual encoder, and image semantic self-attention learning is performed. The formula is expressed as:

[0076] .

[0077] where, denote the image feature after self-attention learning, denote the linear layer, is an activation function.

[0078] Secondly, the text prompt feature obtained by the pre-trained text encoder is First, self-optimization learning is performed, and the formula is as follows:

[0079] .

[0080] wherein, denote the text prompt feature after self-optimization learning.

[0081] Finally, the and are subjected to dimension alignment operation and then subjected to element-wise weighted fusion operation:

[0082] .

[0083] wherein, is the optimized learnable text prompt, and denote the dimension change and dimension expansion operations on the feature vector, denote the element-wise multiplication operation.

[0084] S23: The learnable text prompt and the refined visual prompt are fused to build a continuous multimodal prompt memory bank.

[0085] In a specific embodiment, although the traditional multi-class anomaly detection method can recognize multiple classes at the same time, it requires all classes to be available at one time for training, which is not practical in real applications. In order to solve the problem of unsupervised multi-modal anomaly detection in the context of continuous learning, this embodiment fuses the learnable text prompt and the refined visual prompt to build a continuous multimodal prompt memory bank. As shown in Figure 4 , a four-tuple structure is used to fuse the task identifier (K), the learnable text prompt (T), the refined visual prompt (V), and the normal feature storage (N), to realize multi-modal feature collaborative optimization and efficient knowledge accumulation. For a sequentially arriving task t, a four-tuple is constructed to dynamically adapt to different tasks and guide the model to perform unsupervised anomaly detection.

[0086] In the task adaptation stage, the task identifier uses farthest point sampling Filtering key embeddings from the frozen pre-trained visual branch ViT hierarchical model. This process builds low-dimensional, high discriminative task representations and stores them in a multimodal cue memory bank for task-agnostic prediction. Since the task identity can not always be available in real-world scenarios, task-agnostic prediction is indispensable. For the training set of task t where is the number of images in task t, denotes the size of a single image, since the task identity contains both textual detail information and high-level information, this embodiment uses ViT extraction of a specific layer instead of the last embedding to extract task-related identity information:

[0087] .

[0088] where, is the number of image block features, denotes the dimension of image features, is the feature extracted by the jth layer of ViT, and j is selected as 5. Define the task set , then the identity of the task identifier K can be expressed as:

[0089] .

[0090] where, is the number of tasks, is the farthest point sampling, is the length of . t represents task t, n is the length of the identity representation, , respectively represent the identity of the current task and the identity of each task stored in the multimodal cue memory bank.

[0091] In the task reasoning phase, for a test image , the identity information is extracted by the frozen pre-trained visual branch ViT hierarchical model , and the highest similarity between and the task identifier is calculated to identify the task identity. In addition, this embodiment uses the fine-tuned pre-trained visual branch ViT hierarchical model to extract normal features and form a normal feature bank. Similar to the task identifier K, the extracted image block-level features are compressed using core set sampling (CSS) to obtain the final normal feature bank , which is used to perform nearest neighbor calculation on the extracted normal features and the normal features F stored in the multimodal cue memory bank to obtain the anomaly score. Define the normal feature set where is the number of normal image block features. Then can be represented as:

[0092]

[0093]

[0094] wherein is the core set sampling, is the number of normal image block features in the core set of normal features is the core set of normal features extracted in the task adaptation stage is the space of the core set of normal features extracted in the task adaptation stage Through continuous iterative optimization of multi-modal prompts, efficient task knowledge accumulation and transfer are achieved. Text prompts focus on semantic-level normal mode modeling, and visual prompts enhance local feature adaptation. Both of them form complementary feature representations through shared task identifiers to achieve cross-modal interaction.

[0095] S24: Utilize the adaptive fusion mechanism based on defect semantic guidance to enhance the segmentation performance in anomaly detection.

[0096] In a specific embodiment, as shown in Figure 5 To fully utilize the complementary advantages of multi-modal information, the embodiment proposes an adaptive fusion mechanism based on defect semantic guidance to enhance the segmentation performance in anomaly detection. To enhance the discriminative robustness of the anomaly score, a dynamic Sigmoid normalization mechanism is designed, which is based on the use of hyperparameters k and b to control the steepness and center position of the Sigmoid function to achieve optimization for the score. Among them, k is set to a fixed value, which is 2, because searching for a and b at the same time will bring a large computational burden. Then, for the hyperparameter , the adaptive update of the parameter is realized through a candidate value search strategy to adapt to different tasks. Specifically, a discrete step set is defined, and in each training iteration, a candidate value set is generated based on the current . Among them, is the step size, denotes the discrete step set. Subsequently, the anomaly score is calculated after normalizing each candidate , and the optimal candidate is selected by evaluating the short-term detection performance on the validation set, and its update rule is as follows:

[0097]

[0098] wherein denotes the current candidate​​​​ corresponding validation set performance indicators.

[0099] Further, the improved normalization function is defined as:

[0100] .

[0101] wherein, represents the preliminary anomaly score.

[0102] By dynamically searching the center position of the function, it can adaptively compress the normal sample score interval, while amplifying the response difference of the abnormal area, so as to realize more accurate anomaly positioning. Finally, the normalized and are respectively up-sampled to the same resolution to generate the anomaly map based on the memory bank and the multi-modal alignment anomaly map . The final anomaly score map is generated through the following dynamic fusion strategy:

[0103] .

[0104] wherein, represents the fusion coefficient, used to control the fusion of the anomaly score maps of the visual branch and the text branch.

[0105] S3: Use the dataset to train the learnable visual cues and the learnable text cues in the unsupervised anomaly detection model.

[0106] As shown in Figure 2 , the anomaly detection model framework is built using the pseudo anomaly image generation module, the refined feature cue module, the text cue design module, the learnable text cue module, the continuous multi-modal cue memory bank and the defect semantic guided adaptive fusion mechanism, and the training dataset is input into the built anomaly detection model for training.

[0107] First, the mean square error is used as the loss function to optimize the learnable text cue:

[0108] .

[0109] wherein, represents the mean square error loss function, represents the output of the model, represents the corresponding label value, is the total number of samples. This method enables the model to align the feature distribution of normal samples with text cues without the need for manual labeling, thereby facilitating the learning of normal patterns.

[0110] Secondly, a structured contrastive loss is introduced to further improve the compactness of visual features in the continuous multi-modal prompt memory, so as to better compress normal features and save storage space. The formula is:

[0111] ,

[0112] ,

[0113] .

[0114] In the given loss function, denotes the feature embedding at position (i, j), and denotes the label of the feature corresponding to the position in the segmentation result generated by the pre-trained Segment Anything Model (SAM), (p, q) represents the two-dimensional position coordinates when the feature map. and The values of and are both 1. Through continuous iterative optimization of multi-modal prompts, effective accumulation and transfer of task knowledge are realized, and the robustness and cross-task generalization ability of unsupervised anomaly detection are improved.

[0115] S4: using the trained unsupervised anomaly detection model to perform anomaly detection on industrial data.

[0116] Embodiment two

[0117] Embodiment two of the present application provides an unsupervised continuous anomaly detection system of multi-modal prompt memory, comprising:

[0118] A data acquisition module configured to acquire an industrial image constituting dataset, and pre-process the industrial image;

[0119] A model construction module configured to construct an unsupervised anomaly detection model, including a visual branch and a text branch, respectively used for extracting visual features and text features, introducing a learnable visual prompt and a learnable text prompt in the unsupervised anomaly detection model, using the visual prompt to guide the extraction of visual features, using the text prompt to guide the extraction of text features, and then using an adaptive fusion mechanism to fuse the visual features and the text features to obtain an anomaly detection result;

[0120] A model training module configured to train the learnable visual prompt and the learnable text prompt in the unsupervised anomaly detection model using the dataset;

[0121] Anomaly detection module configured to perform anomaly detection on the industrial data using the trained unsupervised anomaly detection model.

[0122] Embodiment three:

[0123] Embodiment three of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is suitable for being loaded and executed by a processor to perform the steps in the unsupervised continuous anomaly detection method of multi-modal prompt memory.

[0124] Embodiment four:

[0125] Embodiment four of the present application provides a computer device, the device comprises:

[0126] A processor adapted to execute a computer program;

[0127] A computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by the processor, the steps in the unsupervised continuous anomaly detection method of multi-modal prompt memory are realized.

[0128] The steps and methods in the above embodiments two, three and four correspond to the embodiment one, and the specific embodiments can refer to the related description part of the embodiment one.

[0129] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of electronic hardware and computer software. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0130] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable device. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data processing device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD) or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0131] The above description is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An unsupervised persistent anomaly detection method based on multimodal cue memory, characterized in that, Includes the following steps: Acquire industrial image datasets and preprocess the industrial images; An unsupervised anomaly detection model is constructed, including a visual branch and a text branch, which are used to extract visual features and text features, respectively. Learnable visual cues and learnable text cues are introduced into the unsupervised anomaly detection model. Visual cues are used to guide visual feature extraction, and text cues are used to guide text feature extraction. Then, an adaptive fusion mechanism is used to fuse visual features and text features to obtain anomaly detection results. The dataset was used to train the unsupervised anomaly detection model with learnable visual cues and learnable text cues. Anomaly detection in industrial data is performed using a trained unsupervised anomaly detection model. The specific steps for constructing an unsupervised anomaly detection model are as follows: A preliminary unsupervised anomaly detection model has been constructed. Introduce learnable visual cues and learnable text cues into the unsupervised anomaly detection model; By fusing learnable text cues and refined visual cues, a continuous multimodal cue memory is built. The task identifier K, learnable text cues T, refined visual cues V, and normal feature storage N are fused through the quadruple structure M=(K,T,V,F) to achieve multimodal feature collaborative optimization and efficient knowledge accumulation. An adaptive fusion mechanism guided by defect semantics is used to enhance segmentation performance in anomaly detection. The dynamic sigmoid normalization function is: Where x represents the initial anomaly score, P(b) represents the performance metric of the validation set corresponding to the current candidate b. These are candidate values ​​for the hyperparameter b. The step size is denoted by k, and the hyperparameter k is a fixed value.

2. The unsupervised persistent anomaly detection method based on multimodal cue memory as described in claim 1, characterized in that, The preprocessed dataset is divided into a training set and a test set. The unsupervised anomaly detection model is trained using the training set and tested using the test set. The training set contains only normal industrial images. To conform to the scenario setting of unsupervised anomaly detection, the industrial image categories in the test set are consistent with those in the training set. The test set contains various types of abnormal and normal industrial images.

3. The unsupervised persistent anomaly detection method based on multimodal cue memory as described in claim 1, characterized in that, The specific steps for initially constructing an unsupervised anomaly detection model are as follows: Anomaly images are generated by adding noise using a pseudo-anomaly image data generation module. Construct visual and text branches, and use them to extract visual and text features respectively.

4. The unsupervised persistent anomaly detection method based on multimodal cue memory as described in claim 3, characterized in that, Both the visual branch and the text branch are composed of pre-trained ViT models.

5. The unsupervised persistent anomaly detection method based on multimodal cue memory as described in claim 1, characterized in that, The specific steps for introducing learnable visual cues and learnable text cues into an unsupervised anomaly detection model are as follows: A refined feature suggestion module is constructed, and a hierarchical and learnable visual suggestion is introduced to guide the unsupervised anomaly detection of visual features; Learnable text prompts are introduced to guide unsupervised anomaly detection of text features. Specifically, a text prompt design module is constructed, a text semantic template containing learnable text prompts is introduced, and a learnable text prompt module is constructed. The learnable text prompts are adaptively optimized by fusing inherent semantic information of the image.

6. The unsupervised persistent anomaly detection method based on multimodal cue memory as described in claim 1, characterized in that, The specific steps for training the unsupervised anomaly detection model using learnable visual and textual cues on the dataset are as follows: Optimize learnable text prompts using mean squared error as the loss function; Introducing structured contrast loss further enhances the compactness of visual features in the continuous multimodal cue memory.

7. An unsupervised persistent anomaly detection system with multimodal cue memory, characterized in that, The unsupervised persistent anomaly detection method using multimodal cue memory as described in any one of claims 1-6 includes: The data acquisition module is configured to acquire a dataset of industrial images and preprocess the industrial images. The model building module is configured to build an unsupervised anomaly detection model, including a visual branch and a text branch, which are used to extract visual features and text features, respectively. Learnable visual cues and learnable text cues are introduced into the unsupervised anomaly detection model. Visual cues are used to guide visual feature extraction, and text cues are used to guide text feature extraction. Then, an adaptive fusion mechanism is used to fuse visual features and text features to obtain anomaly detection results. The model training module is configured to train the unsupervised anomaly detection model on learnable visual cues and learnable text cues using a dataset. The anomaly detection module is configured to use a trained unsupervised anomaly detection model to detect anomalies in industrial data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the unsupervised persistent anomaly detection method of multimodal cue memory as described in any one of claims 1-6.

9. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the unsupervised persistent anomaly detection method of multimodal cue memory as described in any one of claims 1-6.

Images