Industrial image anomaly detection method based on multi-modal large model

Abstract

The invention discloses an industrial image anomaly detection method based on a multi-modal large model, belongs to the field of industrial anomaly detection, and aims to solve the problems of poor expandability and adaptability and weak capability of identifying tiny anomalies in an image when the multi-modal large model is used for industrial anomaly detection. The method comprises the steps of 1, constructing a knowledge understanding training task based on an industrial anomaly detection data set, 2, predicting an anomaly graph by using an expert-guided image sampling module, extracting N regions of interest, inputting the N regions of interest and an input original graph into a visual encoder together to extract visual features, and obtaining a knowledge understanding training task; outputting a visual code which can be identified by the large language model; step 3, constructing an industrial anomaly detection instruction set of visual language alignment; 4, performing fine adjustment on the multi-modal large model by utilizing a knowledge understanding training task and an industrial anomaly detection instruction set; and 5, carrying out anomaly detection on the industrial product by utilizing the trained multi-modal large model.

Description

Technical Field

[0001] The invention relates to an industrial image anomaly detection method based on a multimodal large model, and belongs to the field of industrial anomaly detection. Background Art

[0002] Vision-based industrial anomaly detection is an important research area in industrial quality inspection and automation. It detects and locates anomalies and defects in industrial products by analyzing their images. Traditional industrial anomaly detection methods enable the model to model the feature distribution of normal samples by learning from a large number of normal samples, and detect the existence of anomalies by quantifying the distance between the test samples and the normal feature distribution. Although these unsupervised learning methods perform well on trained categories, they are difficult to adapt to the launch of various new products and changing detection needs. When the manufacturing process is updated or the production line changes, traditional methods must re-collect a large number of normal samples and retrain the model.

[0003] With the development of multimodal large models, they have demonstrated strong zero-shot / few-shot generalization and instruction-following capabilities in many fields. This allows multimodal large models to naturally perform zero-shot / few-shot detection on different products, and to obtain updated manufacturing process information and production line information to adapt to product updates. This makes anomaly detection models based on multimodal large models a possible solution to the above problems. The current large-model-based industrial anomaly detection algorithm introduces traditional industrial anomaly detection models as visual experts, and initially realizes zero-shot / few-shot detection in the field of industrial anomaly detection, but fails to have the ability to adapt to manufacturing process and production line updates.

[0004] First, the existing industrial anomaly detection algorithms (zero-shot / few-shot detection) based on multimodal large models are only trained using traditional question-answering tasks and image description tasks. This training method allows the multimodal large model to learn to perform binary anomaly detection tasks. However, this training method makes the multimodal large model forget its powerful reasoning ability. When using text-based manufacturing process and production line information as additional prompts, it will lead to an abnormal decrease in anomaly detection capabilities. This makes the existing model have low detection accuracy when facing new types of anomalies caused by production line updates and manufacturing process changes. Therefore, how to reactivate the ability of the multimodal large model to reason using production process information so that it can adapt to changes in production lines and manufacturing processes is an urgent problem to be solved.

[0005] At the same time, the tasks performed by general multimodal large models usually require attention to the content of the entire image. A current solution is to retain a low-resolution overall image while evenly dividing the original image into multiple high-resolution sub-images as additional input. This method can significantly improve the visual perception ability of general multimodal large models. However, such an approach is inefficient for anomaly detection. Because the anomaly may only occupy 3% or even less than 1% of the entire image, inputting a large number of normal features will reduce the model's perception of a small number of abnormal areas. Therefore, a solution is needed to guide the multimodal large model to pay additional attention to potential abnormal areas.

[0006] In summary, the existing zero-sample / small-sample anomaly detection algorithms of large multimodal models are difficult to adapt to new products; they have weak ability to adapt to changes in production lines and production processes, and poor scalability and adaptability; and due to the lack of effective guidance, they have weak ability to identify tiny anomalies in images. Summary of the invention

[0007] In order to solve the above-mentioned problems existing in industrial anomaly detection using a multimodal large model, the present invention provides an industrial image anomaly detection method based on a multimodal large model.

[0008] The industrial image anomaly detection method based on a multimodal large model of the present invention includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps:

[0009] Step 1: construct a knowledge understanding training task based on the industrial anomaly detection dataset, wherein the knowledge understanding training task includes two types: a multimodal knowledge understanding task based on the production process and a plain text knowledge understanding task based on the synthesis of the production process;

[0010] Step 2: Use the expert-guided image sampling module to predict the anomaly map and extract N regions of interest. Figure 1 It is input into the visual encoder to extract visual features and output a visual encoding that can be recognized by the large language model;

[0011] Step 3: Build a visual language aligned industrial anomaly detection instruction set;

[0012] Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set;

[0013] Step 5: Use the trained multimodal large model to perform anomaly detection on industrial products:

[0014] The visual image of the industrial product is input, and the expert-guided image sampling module outputs the visual features. The visual features are embedded with additional text prompts and instructions such as the production process as a whole sequence, and input into the trained multimodal large model. The large model gives a binary judgment or a detailed anomaly description and analysis based on the text instructions, thereby completing industrial anomaly detection.

[0015] Preferably, the multimodal knowledge understanding task based on the production process in step 1 is constructed as follows: using a large language model to generate a corresponding thought chain according to samples provided by an industrial anomaly detection dataset, wherein the information of the samples includes the name, picture, production process and manufacturing process of the industrial product, as well as the anomalies and defects that exist;

[0016] According to the abnormal annotation in the sample, there are two thought chain generation schemes:

[0017] Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains;

[0018] Solution 2: For data with only rough labels of abnormal types, generate an inspection list according to the production process as a thinking chain.

[0019] Preferably, the pure text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: a multimodal large model and a large language model generate product descriptions based on some industrial product prototypes collected on the Internet, and augment their types, details, and existing anomalies, and generate industrial anomaly detection answers in the form of a thinking chain based on the product description and the corresponding production process.

[0020] Preferably, the process of extracting visual features by the expert-guided image sampling module in step 2 comprises the following steps:

[0021] Step 21, the input image is first processed by a visual expert to obtain a spatial anomaly map;

[0022] Step 22: Mask the abnormal image predicted by the visual expert, leaving only 10% of the pixels with the highest abnormal scores in the image, and extract N regions of interest using a topological structure analysis algorithm;

[0023] Step 23, determining N bounding boxes according to the N regions of interest in step 22, and intercepting and copying corresponding areas of the input image according to the bounding boxes as N potential abnormal areas;

[0024] Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual features of this image.

[0025] Preferably, the topology structure analysis algorithm in step 22 is an opencv findContours algorithm.

[0026] Preferably, during training, when the number N of regions of interest extracted from the manually annotated segmentation map is less than 4, M regions of interest with random sizes and positions are randomly added as augmentations, and N+M≤4 is satisfied; during testing, the number and positions of the regions of interest are determined from the anomaly map predicted by the visual expert according to the topological structure analysis algorithm.

[0027] Preferably, the process of step 3 of constructing a visual-language aligned industrial anomaly detection instruction set is: based on the anomaly description manually annotated based on the industrial anomaly detection data set, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; the instruction set includes two tasks: a judgment task that only determines whether the product is defective and outputs a description of the product, including the product's attributes and the abnormal appearance and the cause of the abnormality.

[0028] Preferably, step 4 trains the multimodal big model on the knowledge understanding task and the visual language alignment task according to the standard supervised big model fine-tuning method; the training is completed in the form of multi-task mixed training, wherein the visual language alignment task enables the multimodal big model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal big model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the production process of the product.

[0029] Beneficial effects of the present invention:

[0030] 1. The present invention proposes to improve the detection capability of large models for new products and new anomalies by introducing industrial production processes. It also proposes fine-tuning of knowledge understanding and designs a data generation pipeline. With the help of industrial anomaly detection data sets and large language models, multimodal data with thought chain reasoning processes are generated, and supervised fine-tuning of multimodal large models is performed, so that the multimodal large models can improve their anomaly detection capabilities with the help of additional production processes and product design knowledge.

[0031] 2. The present invention proposes an image sampling module guided by visual experts, which selects potential abnormal areas through the guidance and suggestions of visual experts to enhance the anomaly detection effect of the multimodal large model. At the same time, this module avoids the multimodal large model from over-relying on the advice of visual experts by adding random interference to normal areas during training, thereby maintaining the anomaly detection ability of the large model itself.

[0032] 3. The present invention proposes a multimodal large model fine-tuning framework for industrial anomaly detection. It uses a detailed visual language alignment dataset in the field of industrial anomaly detection, which contains a variety of product and industrial abnormal defects and corresponding text descriptions and explanations, to effectively enhance the large model's perception of product and abnormal attributes, and provide a basis for improving anomaly detection performance through reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a multi-modal large model framework diagram for industrial image anomaly detection of the present invention;

[0034] Figure 2 It is a flowchart for fine-tuning a large multimodal model;

[0035] Figure 3 It is a schematic diagram of the principle of the expert-guided latent region perception module;

[0036] Figures 4 to 6 is a schematic diagram of the knowledge understanding fine-tuning data generation pipeline, where Figure 4 It is a schematic diagram of the process of generating answers for coarse label samples in multimodal data; Figure 5 It is a schematic diagram of the sample answer generation process with image description task annotations in multimodal data; Figure 6 It is a schematic diagram of the process of augmentation and answer generation of plain text samples. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0040] Specific implementation method 1: The following is combined Figures 1 to 6 To illustrate this embodiment, the industrial image anomaly detection method based on the multimodal large model described in this embodiment can be fine-tuned by the method provided by the present invention. The multimodal large model can not only detect defects and anomalies in industrial products through product images, but also improve the detection accuracy by adding the production process information of the products. It can also analyze the possible anomalies and defects of the products through instructions, thereby providing a reference for possible problems that may arise in the actual production process.

[0041] The multimodal large model includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps:

[0042] Step 1: construct a knowledge understanding training task based on the industrial anomaly detection dataset, wherein the knowledge understanding training task includes two types: a multimodal knowledge understanding task based on the production process and a plain text knowledge understanding task based on the synthesis of the production process;

[0043] Step 2: Use the expert-guided image sampling module to predict the anomaly map and extract N regions of interest. Figure 1 It is input into the visual encoder to extract visual features and output a visual encoding that can be recognized by the large language model;

[0044] Step 3: Build a visual language aligned industrial anomaly detection instruction set;

[0045] Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set;

[0046] Step 5: Use the trained multimodal large model to perform anomaly detection on industrial products:

[0047] The visual image of the industrial product is input, and the expert-guided image sampling module outputs the visual code; the visual code is embedded with additional text prompts and instructions such as the production process as a whole sequence, and input into the trained multimodal large model. The large model gives a binary judgment or a detailed anomaly description and analysis based on the text instructions, thereby completing industrial anomaly detection.

[0048] The multimodal knowledge understanding task based on the production process in step 1 is constructed as follows: using a large language model to generate corresponding thought chains based on samples provided by the industrial anomaly detection dataset. The sample information includes the name, picture, production process and manufacturing process of the industrial product, as well as the anomalies and defects that exist;

[0049] According to the abnormal annotation in the sample, there are two thought chain generation schemes:

[0050] Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains;

[0051] Solution 2: For data with only rough labels of abnormal types, generate an inspection list according to the production process as a thinking chain.

[0052] The pure text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: the multimodal large model generates product descriptions based on some industrial product prototypes collected on the Internet, and augments their types, details, and existing anomalies, and generates industrial anomaly detection answers in the form of a thinking chain based on the product description and the corresponding production process.

[0053] Step 2 The process of extracting visual features by the expert-guided image sampling module includes the following steps:

[0054] Step 21, the input image is first processed by a visual expert to obtain a spatial anomaly map;

[0055] Step 22: Mask the abnormal image predicted by the visual expert, leaving only 10% of the pixels with the highest abnormality scores in the image, and use the topological structure analysis algorithm to extract N regions of interest, N≤4;

[0056] During training, when the number of ROIs N extracted from the manually annotated segmentation map is less than 4, M ROIs of random size and position are randomly added as augmentations, and N+M≤4 is satisfied; during testing, the number and position of ROIs are determined from the anomaly map predicted by visual experts according to the topological structure analysis algorithm.

[0057] The topological structure analysis algorithm is the opencv findContours algorithm.

[0058] Step 23, determining N bounding boxes according to the N regions of interest in step 22, and intercepting and copying corresponding areas of the input image according to the bounding boxes as N potential abnormal areas;

[0059] Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual encoding of the image.

[0060] The process of step 3 to construct a visual-language aligned industrial anomaly detection instruction set is as follows: based on the anomaly description manually annotated based on the industrial anomaly detection dataset, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; the instruction set includes two tasks: a judgment task that only determines whether the product is defective and outputs a description of the product, including the product's attributes and the appearance of the anomaly as well as the cause of the anomaly.

[0061] Step 4 trains the multimodal big model on the knowledge understanding task and the visual language alignment task according to the standard supervised big model fine-tuning method; the training is completed in the form of multi-task mixed training, in which the visual language alignment task enables the multimodal big model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal big model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the production process of the product.

[0062] See also Figure 1 There are two ways of training or testing the multimodal large model of the present invention: zero sample and few sample. When zero sample is used, only one expert-guided image sampling module is used to input the image to be tested. The image to be tested is cut under the guidance of experts to generate multiple high-resolution potential abnormal area maps with high probability of including abnormal information, together with the original Figure 1 The image is processed by the visual encoder and then output by the multi-layer perceptron to be recognized by the large language model. When using few samples, two expert-guided image sampling modules are used, one of which is used to process the image to be tested and the other provides a reference image without abnormalities, making the training or test results more accurate.

[0063] See also Figure 3 , providing a preferred implementation mode, the implementation mode of the potential abnormal area perception module based on visual expert guidance in step 2 includes:

[0064] Step 211: The input image is first processed by a visual expert to obtain a spatial anomaly map. The visual expert here can use a traditional zero-sample / small-sample industrial anomaly detection algorithm. In this embodiment, MuSc is used as the visual expert;

[0065] Step 212: According to the spatial anomaly map given in step 211, select up to 4 regions of interest as potential anomaly regions. The selection method is to retain the pixels with the largest 10% score after standardizing the spatial anomaly map, and use the topological structure analysis algorithm (opencv findContours) to extract the connected areas. If the number of regions exceeds 4, only the four with the largest area are selected;

[0066] Step 213: Determine N bounding boxes based on the region of interest selected in step 212, where the size of the bounding box is determined by the visual encoder of the multimodal large model. In this embodiment, the CLIP model is used as the visual encoder, so a fixed size of 336x336 is used. The bounding box must include the entire potential abnormal area and cannot exceed the image boundary. The N bounding boxes can overlap each other;

[0067] Step 221: According to the N bounding boxes selected in step 213, the corresponding area of ​​the input image is intercepted and copied. It should be noted that the input image at this time should maintain the original size and resolution and has not been processed in any way other than reading in.

[0068] Step 222: After the capture is completed, the input image should be scaled according to the input size required by the visual encoder of the multimodal large model. The original image and the N regions captured in step 221 are input into the visual encoder together to obtain the visual features of the input image and its potential abnormal regions.

[0069] In this preferred embodiment, by predicting potential abnormal areas with the help of visual experts, the local areas intercepted according to the guidance of visual experts have a higher probability of including abnormal areas, which can help the multimodal large model focus on possible abnormalities and reduce interference from irrelevant backgrounds. In addition, by cropping the original input image, these local potential abnormal areas can be input into the model with a higher resolution than the full image, thereby providing more opportunities to discover minor abnormalities. At the same time, since the visual expert is only responsible for giving possible abnormal areas, the possible prediction errors themselves will not lead to serious misjudgments of the large model. The large model still relies on input images and text instructions to determine whether industrial products have abnormalities.

[0070] See also Figures 4 to 6 , providing a preferred implementation mode, the implementation mode of using industrial anomaly detection data to construct a multimodal knowledge understanding training task in step 1 includes:

[0071] Step 111: Constructing a multimodal knowledge understanding training task needs to be based on an industrial anomaly detection dataset. The knowledge understanding training task is in the form of question-answering, where the question is mainly to determine whether the industrial product in the picture is abnormal. The production process information of the corresponding product is generated by the large language model based on the product picture or product description. There are two ways to generate the supervised answer, which is generated by steps 112 and 113 respectively, depending on whether there is a detailed anomaly description text annotation.

[0072] Step 112: For industrial anomaly detection data with only defect type annotations, the multimodal knowledge understanding training task needs to use a large language model to generate a corresponding inspection list based on the defect type and the corresponding production process information;

[0073] Step 113: For industrial anomaly detection data with detailed anomaly description text annotations, the large language model can be directly used to generate a corresponding thought chain answer based on the anomaly description;

[0074] The main benefit of the knowledge understanding task built based on industrial anomaly detection data is that it helps the multimodal big model align the visual features of the anomaly with the descriptive text of the process that caused the anomaly in the production process information, thereby activating the knowledge of the multimodal big model itself, so that the anomaly can be judged more accurately with the help of the production process information.

[0075] See also Figure 4 , further providing a preferred implementation mode, the implementation mode of using industrial anomaly detection data to construct a plain text knowledge understanding training task in step 1 includes:

[0076] Step 121: The pure text knowledge comprehension training task also adopts the form of question-answering task. The main problem is to determine whether there is an anomaly in the text description of the image. To build a pure text knowledge training task, you first need to collect pictures of various common industrial product prototypes and generate corresponding picture descriptions through a multimodal large model.

[0077] Step 122: Based on the generated prototype description, the large language model is used to augment the descriptions of these products, mainly including product type augmentation and exception augmentation. Product type augmentation augments the prototype description into different subcategories of the same product to increase the diversity of the data. Abnormal augmentation augments the abnormality by adding abnormal-related descriptions to the product description to augment the abnormality. The augmented product description has a high diversity;

[0078] Step 123: Use the large language model to generate the corresponding production process according to the augmented product description, thereby combining the product description and the production process as part of the question in the question-answering task; at the same time, the large language model generates the corresponding answer according to the product description and the production process, thereby forming a complete plain text knowledge understanding task;

[0079] The plain text knowledge comprehension training task in this preferred embodiment is also in the form of a question-answering task like the multimodal knowledge comprehension training task. The difference is that the plain text knowledge comprehension training mainly focuses on guiding the model to use information from the production process. Therefore, the plain text knowledge comprehension training task needs to cover a variety of industrial products and a variety of abnormal situations, so that the model can learn the abnormalities generated by the production process information for the same process, and then judge the abnormalities of industrial products based on the production process information.

[0080] Principle analysis: This invention applies the multimodal large model to the field of industrial anomaly detection by adding an image encoder module based on visual experts and constructing different knowledge understanding training tasks. It has the ability to infer possible anomalies from the production process and apply them to anomaly detection. The guidance of visual experts on potential abnormal areas can enable the model to pay more attention to possible abnormal areas, and the higher resolution enables the model to detect minor anomalies. After knowledge understanding training in the field of industrial anomaly detection, the trained multimodal large model can complete the task of industrial anomaly detection, and can achieve better anomaly detection results based on the analysis of production process information and good product features and abnormal feature recognition capabilities.

[0081] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. An industrial image anomaly detection method based on a multimodal large model, characterized in that: The multimodal large model includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps: Step 1: construct a knowledge understanding training task based on the industrial anomaly detection dataset, wherein the knowledge understanding training task includes two types: a multimodal knowledge understanding task based on the production process and a plain text knowledge understanding task based on the synthesis of the production process; Step 2: Use the expert-guided image sampling module to predict the abnormal image and extract N regions of interest. Input the N regions of interest together with the input original image into the visual encoder to extract visual features and output the visual encoding that can be recognized by the large language model. Step 3: Build a visual language aligned industrial anomaly detection instruction set; Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set; Step 5: Use the trained multimodal large model to perform anomaly detection on industrial products: The visual image of the industrial product is input, and the expert-guided image sampling module outputs the visual features. The visual features are embedded with additional text prompts and instructions such as the production process as a whole sequence, and input into the trained multimodal large model. The large model gives a binary judgment or a detailed anomaly description and analysis based on the text instructions, thereby completing industrial anomaly detection.

2. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: The multimodal knowledge understanding task based on the production process in step 1 is constructed as follows: using a large language model to generate corresponding thought chains based on samples provided by the industrial anomaly detection dataset. The sample information includes the name, picture, production process and manufacturing process of the industrial product, as well as the anomalies and defects that exist; According to the abnormal annotation in the sample, there are two thought chain generation schemes: Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains; Solution 2: For data with only rough labels of abnormal types, generate an inspection list according to the production process as a thinking chain.

3. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1 or 2, characterized in that: The pure text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: the multimodal large model generates product descriptions based on some industrial product prototypes collected on the Internet, and augments their types, details, and existing anomalies, and generates industrial anomaly detection answers in the form of a thinking chain based on the product description and the corresponding production process.

4. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: Step 2 The process of extracting visual features by the expert-guided image sampling module includes the following steps: Step 21, the input image is first processed by a visual expert to obtain a spatial anomaly map; Step 22: Mask the abnormal image predicted by the visual expert, leaving only 10% of the pixels with the highest abnormality scores in the image, and use the topological structure analysis algorithm to extract N regions of interest, N≤4; Step 23, determining N bounding boxes according to the N regions of interest in step 22, and intercepting and copying corresponding areas of the input image according to the bounding boxes as N potential abnormal areas; Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual features of this image.

5. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 4, characterized in that: During training, when the number of ROIs N extracted from the manually annotated segmentation map is less than 4, M ROIs with random positions and sizes are randomly added as augmentations, and N+M≤4 is satisfied; during testing, the number and positions of ROIs are determined from the anomaly map predicted by visual experts according to the topological structure analysis algorithm.

6. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 4, characterized in that: The topology structure analysis algorithm in step 22 is the opencv findContours algorithm.

7. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: The process of step 3 to construct a visual-language aligned industrial anomaly detection instruction set is as follows: based on the anomaly description manually annotated based on the industrial anomaly detection dataset, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; the instruction set includes two tasks: a judgment task that only determines whether the product is defective and outputs a description of the product, including the product's attributes and the appearance of the anomaly as well as the cause of the anomaly.

8. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: Step 4 trains the multimodal big model on the knowledge understanding task and the visual language alignment task according to the standard supervised big model fine-tuning method; the training is completed in the form of multi-task mixed training, in which the visual language alignment task enables the multimodal big model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal big model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the production process of the product.

Images