Industrial product anomaly detection method, system, equipment and medium

Abstract

The invention discloses an industrial product anomaly detection method, system and device and a medium. The method comprises the steps of obtaining text description, an industrial product image and an auxiliary data set; performing local text generation processing on the industrial product image and the auxiliary data set to obtain a local text prompt; performing feature extraction processing on the text description and the industrial product image according to the local text prompt to obtain text features and image features; performing global text generation processing according to the image features to obtain a global text prompt; splicing the text features according to the global text prompt to obtain a text embedding vector; performing multi-scale space attention fusion processing on the image features to obtain a patch embedding vector; and performing anomaly positioning and detection processing according to the patch embedding vector and the text embedding vector to obtain an anomaly detection result. The embodiment of the invention can improve the anomaly detection efficiency, and can be widely applied to the technical field of anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection, and particularly relates to an industrial product anomaly detection method, system, device and medium. BACKGROUND

[0002] The anomaly detection step of an industrial product is crucial. Through anomaly detection, substandard and defective products can be effectively prevented from flowing into the normal production process, and problems in the use process can be prevented. In related technologies, an industrial anomaly detection method directly realizes anomaly detection by training a model with a large number of normal and abnormal samples. However, in actual applications, it is found that abnormal samples of different industrial products are very scarce in an industrial environment, and the scarcity of labeled data reduces the detection accuracy, affecting the anomaly detection efficiency of the industrial product. In summary, the technical problems existing in related technologies need to be improved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide an industrial product anomaly detection method, system, device and medium, which can improve the accuracy of anomaly detection.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application provides an industrial product anomaly detection method, which comprises the following steps:

[0005] Obtaining a text description, an industrial product image and an auxiliary data set;

[0006] Performing local text generation processing on the industrial product image and the auxiliary data set to obtain a local text prompt;

[0007] Performing feature extraction processing on the text description and the industrial product image according to the local text prompt to obtain text features and image features;

[0008] Performing global text generation processing according to the image features to obtain a global text prompt;

[0009] Performing splicing processing on the text features according to the global text prompt to obtain a text embedding vector;

[0010] Performing multi-scale spatial attention fusion processing on the image features to obtain a patch embedding vector;

[0011] Performing anomaly positioning and detection processing on the patch embedding vector and the text embedding vector to obtain an anomaly detection result.

[0012] In some embodiments, the industrial product image and the auxiliary data set are input into a local text generation module, and the local text generation module comprises a first image encoder and a linear layer;

[0013] According to the first image encoder, image recognition is performed on the industrial product image and the auxiliary data set to obtain recognition features.

[0014] According to the linear layer, the recognition features are subjected to dimension transformation processing, and the local text prompt is output.

[0015] In some embodiments, the feature extraction processing of the text description and the industrial product image according to the local text prompt respectively obtains text features and image features, including the following steps:

[0016] The text description and the industrial product image are input into a multi-modal pre-training model, and the multi-modal pre-training model includes a second image encoder and a text encoder.

[0017] According to the text encoder, text feature extraction processing is performed on the text description, and the local text prompt is fused with the extracted features to obtain the text features.

[0018] According to the second image encoder, visual feature extraction processing is performed on the industrial product image, and the local text prompt is fused with the extracted visual features to obtain the image features.

[0019] In some embodiments, the global text generation processing according to the image features obtains a global text prompt, including the following steps:

[0020] The image features are input into a global text generation module, and the image features are subjected to text embedding space mapping processing by the global text generation module to obtain a mapping vector.

[0021] According to the mapping vector, image category information replacement processing is performed on the text description to obtain the global text prompt.

[0022] In some embodiments, the multi-scale spatial attention fusion processing of the image features obtains a patch embedding vector, including the following steps:

[0023] The image features are subjected to linear dimension conversion processing to obtain converted features.

[0024] The converted features are subjected to multi-scale convolution processing to obtain multi-scale features.

[0025] The multi-scale features are subjected to fusion processing to obtain fused features.

[0026] The fused features are subjected to self-attention calculation processing to obtain the patch embedding vector.

[0027] In some embodiments, the anomaly positioning and detection processing according to the patch embedding vector and the text embedding vector to obtain an anomaly detection result comprises the following steps:

[0028] According to the cosine similarity calculation processing of the patch embedding vector and the text embedding vector, a first anomaly score is obtained;

[0029] The first anomaly score is converted to obtain an anomaly map;

[0030] The anomaly map is subjected to multi-level aggregation processing to obtain a pixel anomaly positioning result;

[0031] The patch embedding vector is subjected to clustering processing to obtain an image embedding vector;

[0032] The cosine similarity calculation processing of the image embedding vector and the text embedding vector is performed to obtain a second anomaly score;

[0033] The second anomaly score is subjected to normalization processing to obtain an image anomaly result;

[0034] The anomaly detection result is obtained according to the pixel anomaly positioning result and the image anomaly result.

[0035] In some embodiments, the clustering processing of the patch embedding vector to obtain an image embedding vector comprises the following steps:

[0036] According to the clustering algorithm, the patch embedding vector is subjected to clustering division processing to obtain a clustering set;

[0037] The average value of the anomaly map position score corresponding to each cluster in the clustering set is calculated to obtain a clustering anomaly score set;

[0038] According to the clustering anomaly score set, the cluster with the highest anomaly score is selected from the clustering set to obtain a target cluster;

[0039] The target cluster is subjected to centroid calculation processing to obtain the image embedding vector.

[0040] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes an industrial product anomaly detection system, which comprises:

[0041] A data acquisition module is configured to acquire a text description, an industrial product image, and an auxiliary data set;

[0042] A local text generation module is configured to perform local text generation processing on the industrial product image and the auxiliary data set to obtain a local text prompt;

[0043] The feature extraction module is configured to perform feature extraction processing on the text description and the industrial product image respectively according to the local text prompt, to obtain text features and image features.

[0044] The global text generation module is configured to perform global text generation processing according to the image features, to obtain a global text prompt.

[0045] The feature splicing module is configured to perform splicing processing on the text features according to the global text prompt, to obtain a text embedding vector.

[0046] The feature fusion module is configured to perform multi-scale spatial attention fusion processing on the image features, to obtain a patch embedding vector.

[0047] The anomaly detection module is configured to perform anomaly positioning and detection processing according to the patch embedding vector and the text embedding vector, to obtain an anomaly detection result.

[0048] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0049] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0050] The embodiments of the present application at least have the following beneficial effects: the present application provides an industrial product anomaly detection method, system, device and medium, which obtains local text prompts by performing local text generation processing on an industrial product image and an auxiliary data set, obtains global text prompts by performing global text generation processing according to image features, can locate anomalies in objects of an unknown class according to global and local deep text prompts, and optimizes defect recognition and segmentation on objects of an unknown class. Moreover, the present application performs multi-scale spatial attention fusion processing on image features, so that the model can adaptively focus on related features at different layers and different spaces, fully utilizes spatial image information of each layer, and improves the anomaly detection efficiency of industrial products. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of an industrial product anomaly detection method provided by the embodiments of the present application;

[0052] Figure 2 is a network structure schematic diagram provided by the embodiments of the present application;

[0053] Figure 3A clustering method provided by an embodiment of the present application is shown in the figure.

[0054] Figure 4 A structure diagram of an industrial product anomaly detection system provided by an embodiment of the present application is shown in the figure.

[0055] Figure 5 A hardware structure diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the embodiments of the present application, but are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0057] It can be understood that the terms “first”, “second”, and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word “if” as used herein can be interpreted as “when” or “upon” or “in response to determining”.

[0058] The terms “at least one”, “multiple”, “each”, “any”, and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0061] 1) Artificial Intelligence (AI), is a cross-disciplinary and emerging discipline based on Computer Science, which is a new technical science of theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. It aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems.

[0062] 2) Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized field that studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structure to continuously improve their performance.

[0063] 3) Image processing is a technology that uses computers to analyze images to achieve desired results. It is also known as image processing. Image processing generally refers to digital image processing. Digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners, etc. The elements of the array are called pixels, and their values are called gray values. Image processing techniques generally include image compression, enhancement and restoration, matching, description and identification.

[0064] The abnormality detection step of industrial products is crucial, the main purpose is to find and locate those abnormal patterns that affect the appearance and function of products, to ensure product quality. Through abnormality detection, it can effectively avoid the flow of substandard and defective products into the normal production process, and prevent problems in the use process.

[0065] In related technologies, manual detection methods are not only time-consuming and labor-intensive, but as industrial automation and intelligentization develop, the proportion of abnormal products gradually decreases, leading to challenges in the efficiency and accuracy of manual detection. With the advancement of digital image processing and artificial intelligence technologies, industrial anomaly detection methods continue to enrich, mainly including unsupervised, supervised and self-supervised three categories. Supervised methods train models directly through a large number of normal and abnormal samples, but abnormal samples are very scarce in industrial environments and difficult to obtain; unsupervised methods use only normal samples to learn their feature distribution and identify abnormal samples that do not conform to the distribution. Although it does not rely on abnormal samples, the effect is usually not as good as supervised methods, and the generalization ability of the model is limited in the face of diverse and unpredictable abnormal types. With the development needs, some products show unknowable characteristics, such as privacy and innovative products, which cannot be specifically trained for products, affecting the accuracy of industrial product anomaly detection.

[0066] Therefore, the embodiment of the present application provides an industrial product anomaly detection method, system, device and medium. The local text prompt is obtained by performing local text generation processing on the industrial product image and the auxiliary data set. The global text prompt is obtained by performing global text generation processing according to the image features. The anomaly in the object of the unknown class can be located according to the global and local deep text prompts, and the defect recognition and segmentation on the object of the unknown class are optimized. Moreover, the model can adaptively focus on the related features on different layers and different spaces by performing multi-scale spatial attention fusion processing on the image features, the spatial image information of each layer is fully utilized, and the anomaly detection efficiency of the industrial product is improved.

[0067] The industrial product anomaly detection method provided by the embodiment of the present application relates to the technical field of anomaly detection. The industrial product anomaly detection method provided by the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a tablet computer, a notebook computer, a desktop computer, and the like, but is not limited to this; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements an industrial product anomaly detection method, and the like, but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0069] Figure 1is an optional flowchart of an industrial product anomaly detection method provided by an embodiment of the present application, Figure 1 The method in the method can include, but is not limited to, steps S101-S107.

[0070] Step S101, obtaining a text description, an industrial product image, and an auxiliary data set;

[0071] Step S102, performing local text generation processing on the industrial product image and the auxiliary data set to obtain a local text prompt;

[0072] Step S103, performing feature extraction processing on the text description and the industrial product image according to the local text prompt to obtain text features and image features;

[0073] Step S104, performing global text generation processing according to the image features to obtain a global text prompt;

[0074] Step S105, performing splicing processing on the text features according to the global text prompt to obtain a text embedding vector;

[0075] Step S106, performing multi-scale spatial attention fusion processing on the image features to obtain a patch embedding vector;

[0076] Step S107, performing anomaly positioning and detection processing on the patch embedding vector and the text embedding vector to obtain an anomaly detection result.

[0077] The steps S101 to S107 shown in the embodiments of the present application can obtain the text description, the industrial product image and the auxiliary data set through target object input, database extracted data and the like. The text description is used for the text of the industrial product question description. For example, the text description can be "A photo of a good object" and "A photo of a bad object". The embodiments of the present application use the word "object" to replace the product category name. The industrial product image is an image obtained by real-time detection or uploading input of the industrial product. The auxiliary data set is a sample that does not contain any category of test data, that is, other image data sets that do not contain any category of industrial product image. The embodiments of the present application use medical anomaly segmentation data as the auxiliary data set. The embodiments of the present application obtain the local text prompt by performing local text generation processing on the industrial product image and the auxiliary data set. The local text prompt is used to let the model focus on the real object and the defect position in the image, rather than the background and the global, so as to enhance the model to focus on the local details and thus improve the feature extraction capability of the model. The embodiments of the present application perform feature extraction processing on the text description and the industrial product image according to the local text prompt, to obtain text features and image features. Then, global text generation processing is performed according to the image features to obtain a global text prompt. The global text prompt can extract prior knowledge of the auxiliary data set, and further generate a text class name description by using the image features. In this way, the training can be performed without knowing the object class name, and the processing effect of data privacy can be improved. The embodiments of the present application perform splicing processing on the global text prompt and the text features to obtain a text embedding vector, and perform multi-scale spatial attention fusion processing on the image features to obtain a patch embedding vector. Finally, abnormal positioning and detection processing are performed on the patch embedding vector and the text embedding vector, and the abnormal detection result can be obtained by comparing the similarity between the patch embedding vector and the text embedding vector. Please refer to Figure 2 , Figure 2 A network structure diagram provided by the embodiments of the present application is shown in the figure. The LTG module represents the local text generation module, the Image Encoder module represents the image encoder, Layer1, Layer2, Layer3 and Layer4 represent each layer in the image encoder, the GTG module represents the global text generation module, the TextEncoder represents the image decoder, the LMS module represents the spatial extraction module, the Linear represents the linear layer, the Multi-convolution represents the multi-scale convolution layer, and the Attention represents the self-attention layer. Abnormal Tests and NormalTests represent the text description of the industrial product, and The text feature is represented, the K-clustering module is represented as (K-clustering), and the anomaly score is represented as AnomalyScore. By inputting the text description, the industrial product image, and the auxiliary data set into the network structure of the embodiment of the application, the anomaly of the industrial product image can be judged.

[0078] In some embodiments, the industrial product image and the auxiliary data set are input into a local text generation module, and the local text generation module includes a first image encoder and a linear layer.

[0079] According to the image recognition processing of the first image encoder on the industrial product image and the auxiliary data set, the recognition feature is obtained.

[0080] According to the dimension transformation processing of the linear layer on the recognition feature, the local text prompt is output.

[0081] In the embodiment of the application, the local text generation module includes a first image encoder and a linear layer. By inputting the industrial product image and the auxiliary data set into the first image encoder for image recognition and feature extraction, the recognition feature can be obtained, and by inputting the recognition feature into the linear layer for dimension transformation processing, the local text prompt can be output. The first image encoder can use a vision transformer (VIT) model. In the embodiment of the application, the image feature generated by the first image encoder is further input into a learnable linear layer, and the linear layer is mainly used to learn the difference between normal and abnormal in the auxiliary data set. The embodiment of the application generates a local text prompt by training the auxiliary data set and inputting a real-time image. The generated text prompt is fused with the visual features of each layer, which can improve the ability of the model to recognize abnormal patterns and locate anomalies. Part of the generated text prompt is trained by the auxiliary data set, and the other part of the text prompt is generated by the input real-time image.

[0082] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the application generates a local text prompt, which can make the model focus on the real object and the defect position in the image, rather than the background and the whole, thereby enhancing the model's attention to local details and improving the model's feature extraction capability.

[0083] In some embodiments, the feature extraction processing of the text description and the industrial product image according to the local text prompt respectively obtains text features and image features, including the following steps:

[0084] The text description and the industrial product image are input into a multi-modal pre-training model, and the multi-modal pre-training model includes a second image encoder and a text encoder.

[0085] The text encoder performs text feature extraction processing on the text description, and then fuses the local text prompts with the extracted features to obtain the text features;

[0086] The second image encoder performs visual feature extraction processing on the industrial product image, and then fuses the local text prompts with the extracted visual features to obtain the image features.

[0087] In this embodiment, by inputting text descriptions and industrial product images into a multimodal pre-trained model, the second image encoder in the multimodal pre-trained model can perform visual feature extraction on the industrial product images, and then fuse the generated local text prompts with the extracted visual features to obtain image features. Simultaneously, the text encoder performs text feature extraction on the text description, and then fuses the local text prompts with the extracted features to obtain text features.

[0088] One of the above technical solutions has the following advantages or beneficial effects: By fusing generated local text prompts with the visual features of the image, the embodiment of this application allows the model to focus on the actual objects and defect locations in the image, rather than the background and the global picture, thus enhancing the model's ability to focus on local details and improve its feature extraction capabilities. Simultaneously, fusing local text prompts with the text features of the text encoder allows the text to acquire information about the local features of the image, improving the model's data processing efficiency.

[0089] In some embodiments, the step of performing global text generation processing based on the image features to obtain a global text prompt includes the following steps:

[0090] The image features are input into the global text generation module, and the global text generation module performs text embedding space mapping processing on the image features to obtain the mapping vector;

[0091] The image category information is replaced by the mapping vector to obtain the global text prompt.

[0092] In this embodiment, the global text generation module employs a small neural network, comprising: a 3x3 convolution, a ReLU activation function, a normalization function, and a linear layer with transformed dimensions. This embodiment maps image features to a text embedding space to obtain a mapping vector, and then replaces the object in the text description with the mapping vector, thereby generating a global text prompt. This global text prompt can obtain image category information from the input image. This embodiment also uses a text encoder to generate normal text features from the text description. and abnormal text features The text prompt feature V generated by the global text generation module I Concatenate the normal text feature And the abnormal text feature Then, the text embedding vector is obtained.

[0093] One of the technical solutions has the following advantages or beneficial effects: the embodiments of the present application can obtain image category information by generating a global text prompt according to an input image, without relying on manually defined text descriptions, thereby improving the efficiency of image recognition in the case of not knowing the object class name.

[0094] In some embodiments, the multi-scale spatial attention fusion processing of the image feature to obtain the patch embedding vector includes the following steps:

[0095] Linear dimension conversion processing is performed on the image feature to obtain a converted feature;

[0096] Multi-scale convolution processing is performed on the converted feature to obtain a multi-scale feature;

[0097] Fusion processing is performed on the multi-scale feature to obtain a fusion feature;

[0098] Self-attention calculation processing is performed on the fusion feature to obtain the patch embedding vector.

[0099] In the embodiments of the present application, the image features extracted by each layer in the second image encoder are subjected to linear layer conversion dimension, and then subjected to multi-scale convolution to obtain information of different spaces and receptive fields, and then the image features subjected to multi-scale feature extraction are fused, and after fusion, the image is blocked into a self-attention module. The self-attention module applies self-attention to the blocked features to enhance local features with richer context information, and the processed window features are recombined into an overall feature map to create strong global associations. Specifically, the embodiments of the present application convert the input features by a linear layer, and then extract features by multi-scale convolution. Initially, the input features with a size of (L, C) (where L = H x W) are reshaped into a two-dimensional feature map (H, W, C), and after feature extraction by multi-scale convolution, they are restored to a two-dimensional feature map (H, W, C) using up-sampling. Then, the features are divided into blocks with a size of h x w, thereby obtaining image blocks with a size of h x w. The size of each block is (h, w, C). After segmentation, the size becomes (H x W, H, W, C). Next, each block is reshaped to (h x w, C) for attention calculation, thereby obtaining a size of (h x w, h x w, C). Wherein, the attention is applied within each window, and the sizes are maintained. Importantly, in order to improve efficiency, the attention of all windows is calculated in parallel. Subsequently, each window is reshaped to (h, w, C) to obtain a total size of (h x w, h, w, C). The processed blocks are recombined into a 2D feature map (H, W, C), which is then reshaped into the original 1D sequence (L, C) to obtain patch embeddings of the image.

[0100] One of the above technical solutions has the following advantages or beneficial effects: the embodiments of the present application fuse the image features by multi-scale spatial attention, so that the model can adaptively focus on the relevant features of different layers and different spaces, fully utilize the spatial image information of each layer, and improve the accuracy of anomaly detection.

[0101] In some embodiments, the anomaly positioning and detection processing according to the patch embedding vector and the text embedding vector to obtain an anomaly detection result comprises the following steps:

[0102] According to the patch embedding vector and the text embedding vector, a cosine similarity calculation process is performed to obtain a first anomaly score;

[0103] The first anomaly score is converted to obtain an anomaly map;

[0104] The anomaly map is subjected to multi-level aggregation processing to obtain a pixel anomaly positioning result;

[0105] The patch embedding vector is subjected to clustering processing to obtain an image embedding vector;

[0106] performing cosine similarity calculation processing on the image embedding vector and the text embedding vector to obtain a second anomaly score;

[0107] performing normalization processing on the second anomaly score to obtain an image anomaly result;

[0108] obtaining the anomaly detection result according to the pixel anomaly positioning result and the image anomaly result.

[0109] In the embodiments of the present application, the anomaly score is derived by measuring the cosine similarity between the patch embedding vector and the text embedding vector to obtain a first anomaly score, wherein the calculation formula of the anomaly score is as follows:

[0110]

[0111] In the formula, cos represents the cosine similarity, is a reshaping and interpolation function. F P represents the patch embedding vector, represents the abnormal text feature, represents the normal text feature. The first anomaly score of the patch embedding vector is converted into an anomaly map in the embodiments of the present application, M∈R H×W , which represents the anomaly score of each pixel position. Next, the anomaly map is obtained from multiple layers, and these anomaly maps are aggregated in a multi-level manner to finally generate a pixel anomaly positioning result. In the training process, the dice loss and the focal loss can be used to optimize the auxiliary data in the embodiments of the present application, so as to optimize the pixel-level anomaly map. After the semantic-rich image embedding vector F I is extracted, the image-level anomaly score S can be calculated using a method similar to the calculation formula of the anomaly score, by calculating F I and the text embedding and , and then performing softmax normalization processing, the image anomaly result can be obtained. In the training process, the focal loss can be used to optimize the image-level anomaly score. Finally, the anomaly detection result can be obtained according to the pixel anomaly positioning result and the image anomaly result.

[0112] In some embodiments, the patch embedding vector is clustered to obtain an image embedding vector, including the following steps:

[0113] performing clustering division processing on the patch embedding vector according to a clustering algorithm to obtain a clustering set;

[0114] performing calculation processing on the average value of the anomaly map position score corresponding to each cluster in the clustering set to obtain a clustering anomaly score set;

[0115] According to the set of cluster anomaly scores, a cluster with the highest anomaly score is selected from the set of clusters to obtain a target cluster;

[0116] A centroid calculation process is performed on the target cluster to obtain the image embedding vector.

[0117] In the embodiments of the present application, in order to improve the performance of image-level anomaly detection (AD), a K-means clustering module is introduced, which obtains an image embedding vector by clustering patch embedding vectors. Please refer to Figure 3 The embodiments of the present application aggregate image patch embeddings that are more likely to represent anomalies, realize the fusion of regional-level information, and thus enhance the robustness of image-level anomaly detection. The processing flow of this module includes three steps: first, the K-means algorithm is used to divide the patch embeddings into K clusters; second, the average value of the corresponding anomaly score of each cluster is calculated to obtain the anomaly score of the cluster; finally, the cluster with the highest anomaly score is selected, the centroid thereof is calculated, and the centroid is taken as a semantic-rich image embedding V M , which encapsulates key information about the most abnormal region.

[0118] Next, combined with specific application examples, the scheme of the embodiments of the present application will be described and explained in detail:

[0119] The embodiments of the present application can be applied to the detection scene of industrial products. The industrial product anomaly detection method of the embodiments of the present application is used for real-time detection of industrial products, and a corresponding computer program can also be written according to the industrial product anomaly detection method of the embodiments of the present application. The electronic device or computer readable storage medium storing the computer program is used for anomaly detection of industrial products. The embodiments of the present application use the Pytorch deep learning framework to build a zero-shot anomaly detection model based on language contrast model anomaly generation and text prompts. The structure of the network model is as follows Figure 2The embodiment of the present application designs a local text generator to generate local text prompts from an auxiliary data set and a real-time input image, fuses the image features of the visual encoder, splices the text features behind the image features, can make the model focus on the real object and defect position in the image rather than the background and the whole, enhances the model's attention to local details to improve the model's feature extraction capability, and fuses the text features of the text encoder to make the text obtain the local feature information of the image. In order to enable the model to be applied to the agnostic class, the embodiment of the present application also designs a global text generator to extract the image features of the encoder, generate global text prompts, fuse the text description, and obtain the image category information. It is particularly important to obtain context information in image segmentation, and it is particularly important to use the image features of each layer extracted in the encoder. A multi-scale processing module is designed to extract the features of each space through convolution of different convolution kernels on the image features of each layer in the image encoder, fully extract the features of each layer through a self-attention layer, and finally introduce a K clustering method to improve the ability of anomaly classification. The embodiment of the present application trains the network model by using an industrial data set and an auxiliary data set, completes the training of the model, and finally tests the effect of the model by using a test set picture. According to the evaluation index, the detection and segmentation effect is evaluated. The evaluation indexes that need to be calculated are: area under the curve (AUROC) and maximum F1 score (max-F1) at the best threshold to evaluate the image level and pixel level anomaly detection (AD) performance. The embodiment of the present application also reports the average performance of the field level in the form of (AUROC, max-F1).

[0120] Please refer to Figure 4 The embodiment of the present application also provides an industrial product anomaly detection system, which can realize the above-mentioned industrial product anomaly detection method. The system comprises:

[0121] A data acquisition module 401 is configured to acquire a text description, an industrial product image and an auxiliary data set;

[0122] A local text generation module 402 is configured to perform local text generation processing on the industrial product image and the auxiliary data set to obtain local text prompts;

[0123] A feature extraction module 403 is configured to perform feature extraction processing on the text description and the industrial product image according to the local text prompts to obtain text features and image features;

[0124] A global text generation module 404 is configured to perform global text generation processing on the image features to obtain global text prompts;

[0125] A feature splicing module 405 is configured to perform splicing processing on the text features according to the global text prompts to obtain a text embedding vector;

[0126] a feature fusion module 406, configured to perform multi-scale spatial attention fusion processing on the image features to obtain a patch embedding vector;

[0127] an anomaly detection module 407, configured to perform anomaly positioning and detection processing on the patch embedding vector and the text embedding vector to obtain an anomaly detection result.

[0128] It can be understood that the contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0129] The present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above industrial product anomaly detection method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0130] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0131] Please refer to Figure 5 , Figure 5 a hardware structure of an electronic device of another embodiment is shown, which includes:

[0132] The processor 501 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0133] The memory 502 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 502 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 502 and called and executed by the processor 501 to implement the industrial product anomaly detection method of the embodiments of the present application.

[0134] The input / output interface 503 is configured to realize information input and output.

[0135] The communication interface 504 is configured to realize communication interaction between the device and other devices, and the communication can be realized in a wired manner (for example, a USB, a network cable, or the like) or in a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0136] The bus 505 is configured to transmit information between various components (for example, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504) of the device.

[0137] The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are communicatively connected to each other in the device through the bus 505.

[0138] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the industrial product anomaly detection method.

[0139] It can be understood that the contents in the above method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically realize the functions of the above method embodiments, and the beneficial effects achieved by the storage medium embodiments are the same as the beneficial effects achieved by the above method embodiments.

[0140] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0141] The industrial product anomaly detection method, system, device, and medium provided by the embodiment of the present application can obtain local text prompts by performing local text generation processing on an industrial product image and an auxiliary data set, obtain global text prompts by performing global text generation processing according to image features, can locate anomalies in objects of an unknown class according to the global and local deep text prompts, and optimize defect recognition and segmentation on the objects of the unknown class. In addition, the embodiment of the present application can make the model adaptively focus on related features on different layers and different spaces by performing multi-scale spatial attention fusion processing on image features, fully utilize the spatial image information of each layer, and improve the anomaly detection efficiency of the industrial product.

[0142] The embodiment of the present application aims at an industrial anomaly detection task, and aims to improve the recognition and segmentation capabilities of anomaly detection, and provides a detection method with high anomaly detection accuracy under zero sample. The current mainstream network cannot achieve high recognition accuracy and high segmentation accuracy under zero sample, and cannot obtain suitable text description for unknown classes; at the same time, the image features extracted by the decoder are not fully utilized, and after the image encoder, the image context information is lost, resulting in insufficient segmentation capability of the model.

[0143] The embodiment of the present application designs a global and local deep text prompt method, which can locate anomalies in any unknown class object by training on limited objects and labeled auxiliary data sets, and understand the normal and abnormal patterns between different objects through labeled auxiliary data set training, and optimize defect recognition and segmentation on unknown class objects. Moreover, the embodiment of the present application proposes a multi-scale spatial attention fusion module, so that the model can adaptively focus on related features at different layers and different spaces, and fully utilize the spatial image information of each layer.

[0144] In addition, since the related method usually selects the maximum value of the anomaly graph as the anomaly score, but this is sensitive to noise prediction. The embodiment of the present application uses K-means to aggregate patch embeddings that are more likely to represent anomalies, thereby aggregating region-level information for robust image-level anomaly detection.

[0145] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the present application, and do not constitute a limitation on the technical solutions provided by the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present application are also applicable to similar technical problems.

[0146] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0147] The system embodiments described above are only schematic, and the units described as separate components can be or can not be physically separated, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0148] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0149] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a recited step or its integral sub-steps or additional steps whether or not readily ascertainable from the description or the like. Further, the words "a" or "an", as used herein in the disclosure and elsewhere, are used indiscriminately and are to be interpreted in the same way, i.e. as meaning "one or more".

[0150] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0151] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between systems or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0153] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0154] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0155] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for detecting anomalies in industrial products, characterized in that, The method includes the following steps: Acquire text descriptions, industrial product images, and auxiliary datasets; Local text generation processing is performed on the industrial product images and the auxiliary dataset to obtain local text prompts; Based on the local text prompts, feature extraction processing is performed on the text description and the industrial product image respectively to obtain text features and image features; Global text generation processing is performed based on the image features to obtain a global text prompt; The text features are concatenated based on the global text prompts to obtain a text embedding vector; The image features are subjected to multi-scale spatial attention fusion processing to obtain patch embedding vectors; Anomaly localization and detection are performed based on the patch embedding vector and the text embedding vector to obtain anomaly detection results.

2. The method according to claim 1, characterized in that, The process of generating local text prompts from the industrial product images and the auxiliary dataset includes the following steps: The industrial product image and the auxiliary dataset are input into the local text generation module, which includes a first image encoder and a linear layer. The first image encoder performs image recognition processing on the industrial product image and the auxiliary dataset to obtain recognition features; The recognition features are subjected to dimensional transformation processing based on the linear layer to output the local text prompt.

3. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the text description and the industrial product image based on the local text prompts to obtain text features and image features includes the following steps: The text description and the industrial product image are input into a multimodal pre-trained model, which includes a second image encoder and a text encoder. The text encoder performs text feature extraction processing on the text description, and then fuses the local text prompts with the extracted features to obtain the text features; The second image encoder performs visual feature extraction processing on the industrial product image, and then fuses the local text prompts with the extracted visual features to obtain the image features.

4. The method according to claim 1, characterized in that, The step of generating global text based on the image features to obtain global text prompts includes the following steps: The image features are input into the global text generation module, and the global text generation module performs text embedding space mapping processing on the image features to obtain the mapping vector; The image category information is replaced by the mapping vector to obtain the global text prompt.

5. The method according to claim 1, characterized in that, The process of performing multi-scale spatial attention fusion on the image features to obtain the patch embedding vector includes the following steps: The image features are subjected to linear dimension transformation to obtain transformed features; The transformed features are subjected to multi-scale convolution processing to obtain multi-scale features; The multi-scale features are fused to obtain fused features; The fused features are subjected to self-attention calculation to obtain the patch embedding vector.

6. The method according to any one of claims 1-5, characterized in that, The step of performing anomaly localization and detection processing based on the patch embedding vector and the text embedding vector to obtain anomaly detection results includes the following steps: The first anomaly score is obtained by performing cosine similarity calculation between the patch embedding vector and the text embedding vector. The first anomaly score is transformed to obtain an anomaly map; The anomaly map is subjected to multi-level aggregation processing to obtain pixel anomaly localization results; Clustering is performed on the patch embedding vectors to obtain image embedding vectors; The image embedding vector and the text embedding vector are subjected to cosine similarity calculation to obtain a second anomaly score; The second anomaly score is normalized to obtain the image anomaly result; The anomaly detection result is obtained based on the pixel anomaly localization result and the image anomaly result.

7. The method according to claim 6, characterized in that, The process of clustering the patch embedding vectors to obtain image embedding vectors includes the following steps: The patch embedding vectors are clustered using a clustering algorithm to obtain a cluster set. The average score of the anomaly map position corresponding to each cluster in the cluster set is calculated to obtain the cluster anomaly score set; Based on the cluster anomaly score set, the cluster with the highest anomaly score is selected from the cluster set to obtain the target cluster; The target cluster is subjected to centroid calculation to obtain the image embedding vector.

8. An industrial product anomaly detection system, characterized in that, The system includes: The data acquisition module is used to acquire text descriptions, images of industrial products, and auxiliary datasets; The local text generation module is used to perform local text generation processing on the industrial product image and the auxiliary dataset to obtain local text prompts; The feature extraction module is used to perform feature extraction processing on the text description and the industrial product image based on the local text prompts, respectively, to obtain text features and image features; A global text generation module is used to perform global text generation processing based on the image features to obtain a global text prompt. The feature concatenation module is used to concatenate the text features according to the global text prompt to obtain a text embedding vector; The feature fusion module is used to perform multi-scale spatial attention fusion processing on the image features to obtain the patch embedding vector; An anomaly detection module is used to perform anomaly localization and detection processing based on the patch embedding vector and the text embedding vector to obtain anomaly detection results.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images