Industrial abnormal image segmentation method and system based on large model

Abstract

The invention discloses an industrial abnormal image segmentation method and system based on a large model, and relates to the technical field of artificial intelligence, and the method comprises the following steps: obtaining an industrial abnormal image; obtaining a text prompt corresponding to the industrial abnormal image based on a multi-modal model; inputting the industrial abnormal image and the text prompt into a mask generator, and generating a plurality of segmented mask images and corresponding confidence scores; processing the segmented mask image based on a self-local sensor so as to adjust a mask encoder, and generating an abnormal segmented mask candidate set based on the adjusted mask encoder; obtaining text features and visual features based on a CLIP text encoder and a CLIP image encoder, and obtaining a final abnormal graph based on the text features and the visual features; according to the industrial abnormal image segmentation method and system, the detection quality and efficiency of the abnormal image are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model-based industrial abnormal image segmentation method and system. Background Art

[0002] Industrial Anomaly Detection (IAD) is a technology applied in the field of manufacturing and production. It is mainly used for defect detection of industrial parts. The goal is to automatically detect potential anomaly types and their locations based on industrial-grade images. Through early identification of anomalies, IAD can reduce losses, improve production efficiency, and ensure product quality. Previous technologies usually use deep convolutional neural networks as image feature extractors, mainly focusing on designing dedicated models for each anomaly category and relying on a large amount of normal image data as the training basis.

[0003] Collecting a large number of training images for each anomaly class is not only challenging but also costly with current technologies. In addition, applying different models for each type of anomaly makes actual industrial scenarios too complicated. These limitations hinder the widespread application of existing industrial anomaly detection (IAD) methods. Given the insufficient data and diverse anomaly types in industrial scenarios, a single model that can adapt to multiple anomaly classes without reference to normal images has become a practical need.

[0004] In recent years, many studies based on Large Vision-Language Models (LVLMs) have demonstrated strong zero-shot visual perception capabilities. These models can quickly adapt to tasks in a variety of categories, even when normal images are scarce or completely lacking. Through carefully designed prompts, they can call on prior knowledge stored in the model to enhance detection capabilities. In addition, these models can be fully applied to downstream tasks with simple natural language descriptions and prompts, such as zero-shot classification using carefully constructed prompts.

[0005] Large-scale vision-language models (LVLMs) are only pre-trained on vision and language datasets from the Internet, lacking the understanding of industrial domain knowledge and sensitivity to local abnormal details in images. The huge domain difference between the pre-training dataset and the industrial anomaly dataset causes the mask generator to generate a large number of irrelevant mask candidate regions. These limitations limit their potential in industrial anomaly detection (IAD) tasks. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a large-model-based industrial abnormal image segmentation method and system, which realizes the task migration of industrial abnormal data sets on the large model and improves the detection quality and efficiency of abnormal images.

[0007] The present invention proposes a large model-based industrial abnormal image segmentation method, which comprises the following steps:

[0008] Step 1: Obtain industrial abnormal images;

[0009] Step 2: Obtain text prompts corresponding to industrial abnormal images based on the multimodal model;

[0010] Step 3: Input the industrial anomaly image and the text prompt into a mask generator to generate an anomaly candidate region set, and input the anomaly candidate region set into the anomaly candidate region generator to generate multiple segmentation mask images and corresponding confidence scores;

[0011] Step 4: Divide each segmentation mask image into multiple sub-regions, map each sub-region into a one-dimensional vector through linear mapping, and combine it with position embedding to generate query Q. At the same time, extract features from the divided image through a mask encoder to generate key K and value V. The sub-weight of each sub-region is calculated based on the QKV self-attention mechanism and averaged. The average value of the sub-weight is greater than the set weight upper limit as the abnormal weight, indicating that the sub-region of the corresponding abnormal candidate region has an abnormality. The abnormal candidate regions that are not related to the abnormality are preliminarily screened out, thereby adjusting the mask encoder, and generating the abnormal segmentation mask candidate set corresponding to all segmentation mask images based on the adjusted mask encoder;

[0012] Step 5: Input the text prompt and the pre-set overall prompt and detail prompt into the CLIP text encoder, and add the output text overall feature and text detail feature to obtain the text feature;

[0013] Step 6: Input the industrial abnormal image and the abnormal segmentation mask candidate set into the CLIP image encoder respectively, and add the output image features and mask features to obtain the visual features;

[0014] Step 7: Calculate the similarity between text features and visual features, and re-correct the confidence score corresponding to each abnormal candidate area in the abnormal segmentation mask candidate set based on the similarity calculation result, and arrange the corrected confidence scores in descending order, take the abnormal candidate areas corresponding to the first K confidence scores and aggregate them to obtain the final abnormal map.

[0015] Furthermore, in step 2, the multimodal model adopts the Blip+ChatGPT method. The industrial abnormal image generates a text description containing nouns related to the abnormal object based on Blip, and ChatGPT performs comparative reasoning based on the received text description and the query sentence provided in advance to generate a target noun T that matches the abnormal concept. o .

[0016] Furthermore, in step 3, the mask generator uses the Grounding-Dino model pre-trained on ImageNet to generate a set of abnormal candidate regions with bounding boxes and their corresponding confidence scores;

[0017] The anomaly region generator is an open-world visual segmentation base model built using SAM to generate pixel-level segmentation mask images.

[0018] Furthermore, in step 4, the sub-weight of each sub-region is calculated and averaged based on the QKV self-attention mechanism, specifically:

[0019] A set number of fixed keys K are configured for each query Q to focus on the small group of attention points around the query point;

[0020] The top MLP of the deformable attention mechanism is removed and a projection layer is applied for dimension expansion to calculate the sub-weights for each sub-region and average the sub-weights of all sub-regions.

[0021] Furthermore, each segmentation mask image is divided into 9 sub-regions, and the weight calculation process of the segmentation mask image is as follows:

[0022] Q 1-9 =Proj.[Pos;p1;...;p9];

[0023] K=V=Proj.(γ clip ·(M c ·Q));

[0024]

[0025] W′=Avg(W 1-9 );

[0026] Among them, Q 1-9 is the query Q of 1 to 9 sub-regions, Pos represents the position embedding, M c is the segmentation mask image, W n is the sub-weight of the nth sub-region, n∈[1,9], i represents the i-th segmentation mask image, I is the total number of segmentation mask images, i.e., industrial anomaly images, Attn. is the self-attention mechanism, Q n is the query Q of the nth sub-region, Ki is the key K of the i-th segmentation mask image, Avg is the average operation, W′ is the weight of the segmentation mask image, that is, the sub-weight average, p1; ...; p9 represent 9 sub-regions respectively, γ clip Represents the CLIP text encoder.

[0027] An industrial abnormal image segmentation system based on a large model includes an acquisition module, a text acquisition module, a mask image generation module, an abnormal segmentation mask candidate set generation module, a text feature generation module, a visual feature generation module and an abnormal image generation module;

[0028] The acquisition module is used to acquire industrial abnormal images;

[0029] The text acquisition module is used to obtain text prompts corresponding to industrial abnormal images based on the multimodal model;

[0030] The mask image generation module is used for inputting the industrial anomaly image and the text prompt into the mask generator to generate an anomaly candidate region set, and inputting the anomaly candidate region set into the anomaly candidate region generator to generate a plurality of segmentation mask images and corresponding confidence scores;

[0031] The abnormal segmentation mask candidate set generation module is used to divide each segmentation mask image into multiple sub-regions, map each sub-region into a one-dimensional vector through linear mapping, and combine it with the position embedding to generate the query Q. At the same time, the divided image is subjected to feature extraction through the mask encoder to generate the key K and value V. The sub-weight of each sub-region is calculated based on the QKV self-attention mechanism and averaged. The average value of the sub-weight is greater than the set weight upper limit as the abnormal weight, indicating that the sub-region of the corresponding abnormal candidate region has an abnormality. The abnormal candidate regions that are not related to the abnormality are preliminarily screened out, thereby adjusting the mask encoder, and generating the abnormal segmentation mask candidate sets corresponding to all segmentation mask images based on the adjusted mask encoder;

[0032] The text feature generation module is used to input the text prompt and the pre-set overall prompt and detail prompt into the CLIP text encoder, and add the output text overall feature and text detail feature to obtain the text feature;

[0033] The visual feature generation module is used to input the industrial abnormal image and the abnormal segmentation mask candidate set into the CLIP image encoder respectively, and add the output image features and mask features to obtain the visual features;

[0034] The anomaly map generation module is used to calculate the similarity between text features and visual features, and based on the similarity calculation results, re-correct the confidence score corresponding to each anomaly candidate area in the anomaly segmentation mask candidate set, and arrange the corrected confidence scores in descending order, take the anomaly candidate areas corresponding to the first K confidence scores and aggregate them to obtain the final anomaly map.

[0035] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned industrial abnormal image segmentation method when executing the computer program.

[0036] A computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the industrial abnormal image segmentation method as described above.

[0037] The advantages of the industrial abnormal image segmentation method and system based on a large model provided by the present invention are that the existing pre-trained multimodal large model is used to complete the abnormal segmentation task. Previously, due to the domain gap between the pre-trained language vision data set and the industrial abnormal data set, it was impossible to directly perform task migration on the large model. In this embodiment, multimodal prompts are used to successfully help the large model release its powerful zero-shot visual perception ability, thereby improving the detection quality and efficiency of abnormal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0039] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0040] In this embodiment, industrial anomaly detection is defined as: identifying and monitoring abnormal conditions or abnormal behaviors in systems, equipment or processes through technical means, so as to take measures early to avoid possible problems. Its goal is to identify patterns or events that are inconsistent with normal behavior from a large amount of data, thereby improving production efficiency and product quality.

[0041] like Figure 1 As shown in the figure, the present invention proposes a large model-based industrial abnormal image segmentation method, which mainly consists of four parts: mask generator, abnormality descriptor, self-local sensor and abnormal visual text aligner. The specific steps are as follows:

[0042] Step 1: Obtain industrial abnormal images;

[0043] Step 2: Obtain text prompts corresponding to industrial abnormal images based on the multimodal model;

[0044] This embodiment realizes the generation of text prompts based on the anomaly descriptor, specifically: the "Blip+ChatGPT" method is adopted as an alternative to large multimodal models. An important advantage of this method is that it does not require retraining, but achieves open-world object detection through pre-trained weights. Specifically, Blip is responsible for receiving abnormal images and completing understanding and generation tasks. It analyzes the image based on the pre-trained weights and generates a text description containing nouns related to the abnormal object. Subsequently, ChatGPT takes over the task and compares and infers by receiving the generated description with the provided query statement. It matches and generates the target noun T that best fits the concept of "abnormal" from the text description. o , that is, text prompt.

[0045] Step 3: Input the industrial anomaly image and the text prompt into a mask generator to generate an anomaly candidate region set, and input the anomaly candidate region set into the anomaly candidate region generator to generate multiple segmentation mask images and corresponding confidence scores;

[0046] Based on the text-guided open set object detection framework, we build a mask generator architecture for visual localization. The architecture uses the Grounding-Dino model pre-trained on ImageNet. Through the text encoder and visual encoder, the model extracts features of abnormal text cues and query objects respectively. These features are used by the cross-modal decoder to generate a set of abnormal candidate regions with bounding boxes and their corresponding confidence scores.

[0047] Subsequently, an open-world visual segmentation base model is constructed using SAM (Segment Anything Model) as an anomaly region generator. This model is also trained on a large-scale image segmentation dataset. Specifically, it takes the bounding box of the anomaly candidate region set as input and passes it to the prompt conditional mask decoder to generate a pixel-level anomaly segmentation mask. Under the conditions of the input image I and the text prompt P, the segmentation mask M can be obtained. c And the corresponding confidence score S:

[0048] M c ,S=Generator(I,P);

[0049] Among them, Generator is a mask generator.

[0050] Step 4: Divide each segmentation mask image into multiple sub-regions, map each sub-region into a one-dimensional vector through linear mapping, and combine it with position embedding to generate query Q. At the same time, extract features from the divided image through a mask encoder to generate key K and value V. The sub-weight of each sub-region is calculated based on the QKV self-attention mechanism and averaged. The average value of the sub-weight is greater than the set weight upper limit as the abnormal weight, indicating that the sub-region of the corresponding abnormal candidate region has an abnormality. The abnormal candidate regions that are not related to the abnormality are preliminarily screened out, thereby adjusting the mask encoder, and generating the abnormal segmentation mask candidate set corresponding to all segmentation mask images based on the adjusted mask encoder;

[0051] This embodiment proposes a self-local perceptron based on an improved self-attention mechanism. Compared with the original QKV self-attention mechanism in Transformer, this embodiment divides the segmented mask image into 9 sub-regions (patches). Each sub-region is mapped to a one-dimensional vector through a linear mapping and combined with positional embeddings to generate the final query Q (query). At the same time, the segmented mask image after division is feature extracted by a mask encoder to generate keys K (keys) and values ​​V (values). Referring to the deformable attention mechanism, this embodiment only assigns a small number of fixed keys K (keys) to each query, focusing on a small group of attention points around the query point. The top MLP layer (multi-layer perceptron) of the deformable attention mechanism is removed, and the projection layer is finally applied for dimensional flattening to calculate the sub-weights and perform an average operation. A higher average value indicates that there are abnormal parts in the sub-regions of the abnormal candidate region, thereby helping the anomaly detection model to preliminarily screen out candidate regions that are not related to the anomaly. Among them, the calculation process of the abnormal weight W′ of a segmentation mask image can be expressed as:

[0052] Q 1-9 =Proj.[Pos;p1;...;p9];

[0053] K=V=Proj.(γ clip ·(M c ·Q));

[0054]

[0055] W′=Avg(W 1-9 );

[0056] Among them, Q 1-9 is the query Q of 1 to 9 sub-regions, Pos represents the position embedding, M c is the segmentation mask image, Wn is the sub-weight of the nth sub-region, n∈[1,9], i represents the i-th segmentation mask image, I is the total number of segmentation mask images, i.e., industrial anomaly images, Attn. is the self-attention mechanism, Q n is the query Q of the nth sub-region, K i is the key K of the i-th segmentation mask image, Avg is the average operation, W′ is the weight of the segmentation mask image, that is, the sub-weight average, p1;...;p9 represent 9 sub-regions respectively, Represents the CLIP text encoder.

[0057] Step 5: Input the text prompt and the pre-set overall prompt and detail prompt into the CLIP text encoder, and add the output text overall feature and text detail feature to obtain the text feature;

[0058] Step 6: Input the industrial abnormal image and the abnormal segmentation mask candidate set into the CLIP image encoder respectively, and add the output image features and mask features to obtain the visual features;

[0059] Step 7: Calculate the similarity between text features and visual features, and re-correct the confidence score corresponding to each abnormal candidate area in the abnormal segmentation mask candidate set based on the similarity calculation result, and arrange the corrected confidence scores in descending order, take the abnormal candidate areas corresponding to the first K confidence scores and aggregate them to obtain the final abnormal map.

[0060] Based on the abnormal visual text aligner, steps five to seven are implemented as follows:

[0061] In order to reduce language ambiguity and make full use of the potential of text features of domain expert knowledge to achieve better detection results, this embodiment proposes a text feature positioning anomaly module. First, the anomaly prompt consists of two parts, namely, detail prompts and overall prompts, where the detail prompts are also used to supplement the initialization prompts in the anomaly descriptor. The detail prompts are designed based on expert knowledge of anomaly patterns of similar products to supplement more specific anomaly details. This prompt method rephrases the task of finding abnormal areas as locating objects with specific abnormal state expressions, which makes the extracted text features better match the pre-trained model and makes better use of the basic model than identifying "anomalies" in the object context. By combining the above two prompts, it is possible to refine the simple "abnormal" prompt into a more specific prompt that describes the abnormal state in more detail, and use the pre-trained CLIP text encoder to extract the text detail feature DF. t :

[0062] DF t =θ clip (dp,T O );

[0063] Among them, dp is the detail prompt, T O is the text prompt, θ clip A pre-trained CLIP text encoder.

[0064] The overall prompt is an overall description of each image in the industrial anomaly dataset. Since the industrial anomaly images in the industrial anomaly dataset are normalized and only differ in the state or quantity of the anomalies, this embodiment uniformly describes the number of industrial product anomalies, the maximum number of abnormal regions, and the size of the abnormal area to make up for the deficiencies caused by the lack of specific attribute recognition of the basic model. Such a design helps to understand the overall expression of the target while focusing on the target object itself. The pre-trained CLIP text encoder is also used to extract the overall text feature OF t :

[0065] OF t =θ clop (op);

[0066] Among them, θ clip is the pre-trained CLIP text encoder and op is the overall prompt.

[0067] Then the text detail feature DF t and the overall text feature OF t Combined to get the text feature T f , as shown below:

[0068] T f =DF t +OF t .

[0069] In the context feature of the text (i.e., text feature T f ), the previously obtained abnormal segmentation mask candidate set is used for further optimization. Since different masks segment the features of different regions, different regions actually contain different feature information. However, the original visual features of CLIP are designed to generate a single feature vector to describe the entire image, which is not suitable for dense prediction at the pixel level. To solve this problem, this embodiment modifies the visual encoder in CLIP to extract features containing information about the occluded area and the surrounding area. Specifically, given the input industrial abnormal image and the abnormal segmentation mask candidate set, the visual features are composed of the overall image features and a new image obtained by the abnormal segmentation mask image only around the mask suggestion area, which are passed to the corresponding Clip image encoders respectively:

[0070]

[0071] Among them, DFv is the mask feature, OF v is the image feature, For Clip image encoder, dv crop is the mask encoder, I is the industrial abnormality image, ov is the image feature, ⊙ is the Hadamard product, m is the number of times the detail features are extracted from the industrial abnormality image, that is, if there are m masks, image features must be extracted m times, and ov is the extracted image feature.

[0072] Then we can extract the “overall + details” visual context feature, namely the visual feature V f , as shown below:

[0073] V f =DF v +OF v .

[0074] Since this embodiment is based on CLIP, where visual features and text features are embedded in a common space, each segmentation mask in the abnormal segmentation mask candidate set has a fixed set of visual language feature combinations. f and visual features V f The score of each abnormal segmentation mask suggestion is obtained by calculating the similarity between the visual features and text features of all mask suggestions. Given the input of image I and reference expression T, this embodiment can calculate the similarity C between the visual features and text features of all mask suggestions m :

[0075] C m =sim(V f ,T f );

[0076] Among them, sim is the cosine similarity calculation.

[0077] Usually, the number of abnormal regions in the inspected object is limited. Therefore, this embodiment selects K candidate regions with the highest confidence scores and uses their average value to perform the final abnormal region detection.

[0078] That is, the score corresponding to a single region in the abnormal segmentation mask candidate set is represented as S m , recalibrated using cosine similarity:

[0079] s=C m ·S m

[0080] M′ C ,S′ C =Top k (M c ,s)

[0081] M′ C ,S ′ C They are the calibrated segmentation masks and the corresponding confidence scores, and then these K candidate regions are aggregated to estimate the final anomaly map A:

[0082]

[0083] Among them, s is the corrected confidence score, Top k (M c ,s) is the abnormal candidate region corresponding to the first K confidence scores, M′ C ,S C ′ is the modified abnormal segmentation mask and the corresponding confidence score, α is the balance coefficient, which is preferably set to 0.1, and the specific value can be adjusted in the experiment.

[0084] When performing anomaly detection on industrial abnormal images, the normal reference image is used as a reference benchmark, thereby facilitating the identification of anomalies from the industrial abnormal images.

[0085] Through more comprehensive industry domain expert knowledge and more precise prompt engineering, the final optimized framework is obtained, which can generate more reliable predictions.

[0086] According to steps one to seven, this embodiment utilizes the existing pre-trained multimodal large model to complete the anomaly segmentation task without any further training. Previously, due to the domain gap between the pre-trained language vision dataset and the industrial anomaly dataset, it was not possible to directly transfer tasks on the large model. In this embodiment, multimodal prompts are used to successfully help the large model unleash its powerful zero-shot visual perception capabilities. By integrating multiple data types from different data sources, large models can generalize from learned patterns to new data and tasks, which enables them to achieve good results in other applications and provides strong technical support for various complex multimodal processing. Provides valuable inspiration for non-adaptive design in the multimodal field.

[0087] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for industrial abnormal image segmentation based on a large model, characterized in that: The steps include: Step 1: Obtain industrial abnormal images; Step 2: Obtain text prompts corresponding to industrial abnormal images based on the multimodal model; Step 3: Input the industrial anomaly image and the text prompt into a mask generator to generate an anomaly candidate region set, and input the anomaly candidate region set into the anomaly candidate region generator to generate multiple segmentation mask images and corresponding confidence scores; Step 4: Divide each segmentation mask image into multiple sub-regions, map each sub-region into a one-dimensional vector through linear mapping, and combine it with position embedding to generate query Q. At the same time, extract features from the divided image through a mask encoder to generate key K and value V. The sub-weight of each sub-region is calculated based on the QKV self-attention mechanism and averaged. The average value of the sub-weight is greater than the set weight upper limit as the abnormal weight, indicating that the sub-region of the corresponding abnormal candidate region has an abnormality. The abnormal candidate regions that are not related to the abnormality are preliminarily screened out, thereby adjusting the mask encoder, and generating the abnormal segmentation mask candidate set corresponding to all segmentation mask images based on the adjusted mask encoder; Step 5: Input the text prompt and the pre-set overall prompt and detail prompt into the CLIP text encoder, and add the output text overall feature and text detail feature to obtain the text feature; Step 6: Input the industrial abnormal image and the abnormal segmentation mask candidate set into the CLIP image encoder respectively, and add the output image features and mask features to obtain the visual features; Step 7: Calculate the similarity between text features and visual features, and re-correct the confidence score corresponding to each abnormal candidate area in the abnormal segmentation mask candidate set based on the similarity calculation result, and arrange the corrected confidence scores in descending order, take the abnormal candidate areas corresponding to the first K confidence scores and aggregate them to obtain the final abnormal map.

2. The industrial abnormal image segmentation method based on a large model according to claim 1 is characterized in that: In step 2, the multimodal model adopts the Blip+ChatGPT method. The industrial abnormal image generates a text description containing nouns related to the abnormal object based on Blip. ChatGPT compares and infers the received text description and the query sentence provided in advance to generate the target noun T that matches the abnormal concept. o .

3. The industrial abnormal image segmentation method based on a large model according to claim 1 is characterized in that: In step 3, the mask generator uses the Grounding-Dino model pre-trained on ImageNet to generate a set of abnormal candidate regions with bounding boxes and their corresponding confidence scores; The anomaly region generator is an open-world visual segmentation base model built using SAM to generate pixel-level segmentation mask images.

4. The industrial abnormal image segmentation method based on a large model according to claim 1 is characterized in that: In step 4, the sub-weights of each sub-region are calculated and averaged based on the QKV self-attention mechanism, specifically: A set number of fixed keys K are configured for each query Q to focus on the small group of attention points around the query point; The top MLP of the deformable attention mechanism is removed and a projection layer is applied for dimension expansion to calculate the sub-weights for each sub-region and average the sub-weights of all sub-regions.

5. The large model-based industrial abnormal image segmentation method according to claim 4 is characterized in that: Each segmentation mask image is divided into 9 sub-regions, and the weight calculation process of the segmentation mask image is as follows: Q 1-9 =Proj.[Pos;p1;...;p9]; K=V=Proj.(γ clip ·(M c ·Q)); W′=Avg(W 1-9 ); Among them, Q 1-9 is the query Q of 1 to 9 sub-regions, Pos represents the position embedding, M c is the segmentation mask image, W n is the sub-weight of the nth sub-region, n∈[1,9], i represents the i-th segmentation mask image, I is the total number of segmentation mask images, i.e., industrial anomaly images, Attn. is the self-attention mechanism, Q n is the query Q, K of the nth sub-region i is the key K of the i-th segmentation mask image, Avg is the average operation, W′ is the weight of the segmentation mask image, that is, the sub-weight average, p1; ...; p9 represent 9 sub-regions respectively, γ clip Represents the CLIP text encoder.

6. An industrial abnormal image segmentation system based on a large model, characterized in that: It includes an acquisition module, a text acquisition module, a mask image generation module, an abnormal segmentation mask candidate set generation module, a text feature generation module, a visual feature generation module and an abnormal map generation module; The acquisition module is used to acquire industrial abnormal images; The text acquisition module is used to obtain text prompts corresponding to industrial abnormal images based on the multimodal model; The mask image generation module is used for inputting the industrial anomaly image and the text prompt into the mask generator to generate an anomaly candidate region set, and inputting the anomaly candidate region set into the anomaly candidate region generator to generate a plurality of segmentation mask images and corresponding confidence scores; The abnormal segmentation mask candidate set generation module is used to divide each segmentation mask image into multiple sub-regions, map each sub-region into a one-dimensional vector through linear mapping, and combine it with the position embedding to generate the query Q. At the same time, the divided image is subjected to feature extraction through the mask encoder to generate the key K and value V. The sub-weight of each sub-region is calculated based on the QKV self-attention mechanism and averaged. The average value of the sub-weight is greater than the set weight upper limit as the abnormal weight, indicating that the sub-region of the corresponding abnormal candidate region has an abnormality. The abnormal candidate regions that are not related to the abnormality are preliminarily screened out, thereby adjusting the mask encoder, and generating the abnormal segmentation mask candidate sets corresponding to all segmentation mask images based on the adjusted mask encoder; The text feature generation module is used to input the text prompt and the pre-set overall prompt and detail prompt into the CLIP text encoder, and add the output text overall feature and text detail feature to obtain the text feature; The visual feature generation module is used to input the industrial abnormal image and the abnormal segmentation mask candidate set into the CLIP image encoder respectively, and add the output image features and mask features to obtain the visual features; The anomaly map generation module is used to calculate the similarity between text features and visual features, and based on the similarity calculation results, re-correct the confidence score corresponding to each anomaly candidate area in the anomaly segmentation mask candidate set, and arrange the corrected confidence scores in descending order, take the anomaly candidate areas corresponding to the first K confidence scores and aggregate them to obtain the final anomaly map.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the industrial abnormal image segmentation method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the industrial abnormal image segmentation method according to any one of claims 1 to 6.

Images