Industrial image anomaly detection method based on multimodal large model

Abstract

A method for detecting anomalies in industrial images based on a multimodal large model belongs to the field of industrial anomaly detection. This invention addresses the problems of poor scalability and adaptability, as well as the weak ability to identify minor anomalies in images, when multimodal large models are used for industrial anomaly detection. The method includes the following steps: 1. Constructing a knowledge understanding training task based on an industrial anomaly detection dataset; 2. Using an expert-guided image sampling module to predict anomaly maps and extract N regions of interest (ROIs); 3. Inputting these N ROIs, along with the original image, into a visual encoder to extract visual features and output visual encodings recognizable by a large language model; 4. Constructing a visual-language aligned industrial anomaly detection instruction set; 5. Fine-tuning the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set; and 6. Using the trained multimodal large model to perform anomaly detection on industrial products.

Description

Technical Field

[0001] The present invention relates to an industrial image anomaly detection method based on a multimodal large model, and belongs to the field of industrial anomaly detection. Background Art

[0002] Vision-based industrial anomaly detection, which detects and locates anomalies and defects in industrial products by analyzing their images, is a key research area in industrial quality inspection and automation. Traditional industrial anomaly detection methods rely on learning from a large number of normal samples to enable the model to model the characteristic distribution of normal samples and detect anomalies by quantifying the distance between test samples and the normal characteristic distribution. While these unsupervised learning methods perform well on trained categories, they struggle to adapt to the launch of new products and evolving detection requirements. When manufacturing processes are updated or production lines change, traditional methods require the collection of a large number of normal samples and retraining of the model.

[0003] With the development of large multimodal models, they have demonstrated strong zero-shot / few-shot generalization and instruction-following capabilities in many fields. This allows large multimodal models to naturally perform zero-shot / few-shot detection on different products and to obtain updated manufacturing process and production line information to adapt to product updates. This makes anomaly detection models based on large multimodal models a possible solution to the above problems. Current large-model-based industrial anomaly detection algorithms use traditional industrial anomaly detection models as visual experts, initially achieving zero-shot / few-shot detection in the field of industrial anomaly detection, but fail to adapt to updates in manufacturing processes and production lines.

[0004] First, existing industrial anomaly detection algorithms (zero-shot / few-shot detection) based on multimodal large models are trained only using traditional question-answering and image description tasks. This training method allows multimodal large models to learn to perform binary anomaly detection tasks. However, this training method causes multimodal large models to forget their powerful reasoning capabilities. When using text-based manufacturing process and production line information as additional prompts, it actually leads to an abnormal decrease in anomaly detection capabilities. This results in low detection accuracy for new types of anomalies caused by production line updates and manufacturing process changes. Therefore, how to reactivate the multimodal large model's ability to reason using production process information so that it can adapt to changes in production lines and manufacturing processes is an urgent problem that needs to be solved.

[0005] At the same time, tasks performed by general-purpose multimodal large models often require attention to the content of the entire image. A current solution is to retain a low-resolution overall image while evenly dividing the original image into multiple high-resolution sub-images as additional input. This approach can significantly improve the visual perception capabilities of general-purpose multimodal large models. However, this approach is inefficient for anomaly detection. Because anomalies may only occupy 3% or even less than 1% of the entire image, inputting a large number of normal features will actually reduce the model's ability to perceive small anomalous areas. Therefore, a solution is needed to guide multimodal large models to pay additional attention to potentially anomalous areas.

[0006] In summary, existing multimodal large-scale model zero-sample / small-sample anomaly detection algorithms are difficult to adapt to new products; they have weak ability to adapt to changes in production lines and production processes, and have poor scalability and adaptability; and due to the lack of effective guidance, they are weak in identifying subtle anomalies in images. Summary of the Invention

[0007] In order to solve the above-mentioned problems existing in industrial anomaly detection using a large multimodal model, the present invention provides an industrial image anomaly detection method based on a large multimodal model.

[0008] The industrial image anomaly detection method based on a multimodal large model described in the present invention includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps:

[0009] Step 1: Construct a knowledge comprehension training task based on the industrial anomaly detection dataset. The knowledge comprehension training task includes two types: a multimodal knowledge comprehension task based on the production process and a plain text knowledge comprehension task based on the synthesis of the production process.

[0010] Step 2: Use the expert-guided image sampling module to predict the anomaly map and extract N regions of interest. Figure 1 It is input into the visual encoder to extract visual features and output a visual encoding that can be recognized by the large language model;

[0011] Step 3: Build a visual language aligned industrial anomaly detection instruction set;

[0012] Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set;

[0013] Step 5: Use the trained multimodal model to detect anomalies of industrial products:

[0014] Visual images of industrial products are input, and an expert-guided image sampling module outputs visual features. These visual features are embedded with additional text prompts and instructions such as production processes as a whole sequence and input into a trained multimodal large model. The large model then gives binary judgments or detailed anomaly descriptions and analysis based on the text instructions, thereby completing industrial anomaly detection.

[0015] Preferably, the multimodal knowledge understanding task based on the production process in step 1 is constructed by using a large language model to generate corresponding thought chains based on samples provided by the industrial anomaly detection dataset. The sample information includes the name, picture, production process and manufacturing process of the industrial product, as well as existing anomalies and defects.

[0016] There are two thought chain generation schemes based on the abnormal annotation conditions in the sample:

[0017] Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains.

[0018] Solution 2: For data with only rough labels of abnormal types, generate a checklist according to the production process as a thinking chain.

[0019] Preferably, the plain text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: a multimodal large model and a large language model generate product descriptions based on some industrial product prototypes collected on the Internet, and augment their types, details, and existing anomalies, and generate industrial anomaly detection answers in the form of a thinking chain based on the product description and the corresponding production process.

[0020] Preferably, the process of extracting visual features by the expert-guided image sampling module in step 2 includes the following steps:

[0021] Step 21: The input image is first processed by a visual expert to obtain a spatial anomaly map;

[0022] Step 22: Mask the anomaly map predicted by the visual expert, leaving only the 10% of pixels with the highest anomaly scores in the map, and use the topological structure analysis algorithm to extract N regions of interest;

[0023] Step 23: determine N bounding boxes based on the N regions of interest in step 22, and intercept and copy corresponding areas of the input image according to the bounding boxes as N potential abnormal areas;

[0024] Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual features of this image.

[0025] Preferably, the topology structure analysis algorithm in step 22 is an opencv findContours algorithm.

[0026] Preferably, during training, when the number N of regions of interest extracted from the manually annotated segmentation map is less than 4, M regions of interest with random sizes and positions are randomly added as augmentations, and N+M≤4 is satisfied; during testing, the number and positions of the regions of interest are determined from the anomaly map predicted by the visual expert according to the topological structure analysis algorithm.

[0027] Preferably, the process of step 3 of constructing a visual-language aligned industrial anomaly detection instruction set is as follows: based on the anomaly description manually annotated based on the industrial anomaly detection dataset, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; the instruction set includes two tasks: a judgment task that only determines whether the product is defective and outputs a description of the product, including the product's attributes, the appearance of the anomaly, and the cause of the anomaly.

[0028] Preferably, step 4 trains the multimodal large model on the knowledge understanding task and the visual language alignment task according to the standard supervised large model fine-tuning method; the training is completed in the form of multi-task hybrid training, wherein the visual language alignment task enables the multimodal large model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal large model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the production process of the product.

[0029] Beneficial effects of the present invention:

[0030] 1. This paper proposes to improve the large-scale model's ability to detect new products and new anomalies by incorporating industrial production processes. It also proposes knowledge understanding fine-tuning and designs a data generation pipeline. Leveraging industrial anomaly detection datasets and large language models, this pipeline generates multimodal data with thought chain reasoning. This supervised fine-tuning of the multimodal large-scale model allows it to leverage additional production process and product design knowledge to enhance anomaly detection capabilities.

[0031] 2. This paper proposes a visual expert-guided image sampling module that uses the guidance of visual experts to select potentially abnormal regions, thereby enhancing the anomaly detection performance of the multimodal large model. Furthermore, this module adds random interference to normal regions during training to prevent the multimodal large model from over-reliance on the visual expert's advice, thereby maintaining the large model's inherent anomaly detection capabilities.

[0032] 3. This paper proposes a multimodal large model fine-tuning framework for industrial anomaly detection. It uses a detailed visual language alignment dataset in the field of industrial anomaly detection, which contains a variety of product and industrial anomaly defects and corresponding text descriptions and explanations. It effectively enhances the large model's perception of product and anomaly attributes, and provides a basis for improving anomaly detection performance through reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a multi-modal large model framework diagram for industrial image anomaly detection in the present invention;

[0034] Figure 2 It is a flowchart for fine-tuning a large multimodal model;

[0035] Figure 3 This is a schematic diagram of the principle of the expert-guided latent region perception module;

[0036] Figures 4 to 6 This is a schematic diagram of the knowledge understanding fine-tuning data generation pipeline, where Figure 4 This is a schematic diagram of the process of generating answers for coarse-labeled samples in multimodal data; Figure 5 This is a schematic diagram of the sample answer generation process with image description task annotations in multimodal data; Figure 6 Schematic diagram of the process of augmenting and generating answers for plain text samples. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0040] Specific implementation method 1: Figures 1 to 6 To illustrate this embodiment, the industrial image anomaly detection method based on the multimodal large model described in this embodiment is fine-tuned by the method provided by the present invention. The multimodal large model can not only detect defects and anomalies in industrial products through product images, but also improve the detection accuracy by adding the production process information of the product. It can also use instructions to enable the model to analyze possible anomalies and defects in the product, thereby providing a reference for problems that may arise in the actual production process.

[0041] The multimodal large model includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps:

[0042] Step 1: Construct a knowledge comprehension training task based on the industrial anomaly detection dataset. The knowledge comprehension training task includes two types: a multimodal knowledge comprehension task based on the production process and a plain text knowledge comprehension task based on the synthesis of the production process.

[0043] Step 2: Use the expert-guided image sampling module to predict the anomaly map and extract N regions of interest. Figure 1 It is input into the visual encoder to extract visual features and output a visual encoding that can be recognized by the large language model;

[0044] Step 3: Build a visual language aligned industrial anomaly detection instruction set;

[0045] Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set;

[0046] Step 5: Use the trained multimodal model to detect anomalies of industrial products:

[0047] A visual image of an industrial product is input, and an expert-guided image sampling module outputs a visual code. The visual code is embedded with additional text prompts and instructions such as production processes as a whole sequence and input into a trained multimodal large model. The large model then gives a binary judgment or a detailed anomaly description and analysis based on the text instructions, thereby completing industrial anomaly detection.

[0048] The multimodal knowledge comprehension task based on the production process in step 1 is constructed by using a large language model to generate corresponding thought chains based on samples provided by the industrial anomaly detection dataset. The sample information includes the name and image of the industrial product, the production process and manufacturing process, and any anomalies and defects.

[0049] There are two thought chain generation schemes based on the abnormal annotation conditions in the sample:

[0050] Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains.

[0051] Solution 2: For data with only rough labels of abnormal types, generate a checklist according to the production process as a thinking chain.

[0052] The pure text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: the multimodal large model generates product descriptions based on some industrial product prototypes collected on the Internet, and augments their types, details, and existing anomalies, and generates industrial anomaly detection answers in the form of a thought chain based on the product description and the corresponding production process.

[0053] Step 2 The process of extracting visual features by the expert-guided image sampling module includes the following steps:

[0054] Step 21: The input image is first processed by a visual expert to obtain a spatial anomaly map;

[0055] Step 22: Mask the anomaly map predicted by the visual expert, leaving only the 10% of pixels with the highest anomaly scores in the map, and use the topological structure analysis algorithm to extract N regions of interest, N ≤ 4;

[0056] During training, when the number of ROIs N extracted from the manually annotated segmentation map is less than 4, M ROIs of random size and position are randomly added as augmentations, and N + M ≤ 4 is satisfied. During testing, the number and position of ROIs are determined from the anomaly map predicted by visual experts according to the topological structure analysis algorithm.

[0057] The topology structure analysis algorithm is the OpenCV findContours algorithm.

[0058] Step 23: determine N bounding boxes based on the N regions of interest in step 22, and intercept and copy corresponding areas of the input image according to the bounding boxes as N potential abnormal areas;

[0059] Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual encoding of this image.

[0060] The process of step 3 to construct a visual-language aligned industrial anomaly detection instruction set is as follows: based on the manually annotated anomaly descriptions of the industrial anomaly detection dataset, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; this instruction set includes two tasks: a judgment task that only determines whether the product is defective and an output description of the product, which includes the product's attributes, the appearance of the anomaly, and the cause of the anomaly.

[0061] Step 4 trains the multimodal large model on the knowledge understanding task and the visual-language alignment task according to the standard supervised large model fine-tuning method; the training is completed in the form of multi-task mixed training, in which the visual-language alignment task enables the multimodal large model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal large model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the product production process.

[0062] See also Figure 1 When the multimodal large model of the present invention is used for training or testing, there are two methods: zero sample and few sample. When zero sample is used, only one expert-guided image sampling module is used to input the image to be tested. The image to be tested is cut under the guidance of experts to generate multiple high-resolution potential abnormal area maps with high probability of including abnormal information, together with the original Figure 1 The image is processed by the visual encoder and then output by the multi-layer perceptron, which can be recognized by the large language model. When using a small number of samples, two expert-guided image sampling modules are used, one of which is used to process the image to be tested and the other to provide a reference image without abnormalities, making the training or testing results more accurate.

[0063] See also Figure 3 , providing a preferred embodiment, the implementation of the potential abnormal area perception module based on visual expert guidance in step 2 includes:

[0064] Step 211: The input image is first processed by a visual expert to obtain a spatial anomaly map. The visual expert here can use a traditional zero-shot / small-shot industrial anomaly detection algorithm. In this embodiment, MuSc is used as the visual expert.

[0065] Step 212: Based on the spatial anomaly map generated in step 211, select up to four regions of interest as potential anomaly areas. The selection method is to normalize the spatial anomaly map and retain the pixels with the largest fraction of 10%. A topology analysis algorithm (opencv findContours) is used to extract connected regions. If the number of regions exceeds four, only the four with the largest areas are selected.

[0066] Step 213: Determine N bounding boxes based on the ROI selected in step 212. The size of the bounding boxes is determined by the visual encoder of the multimodal large model. In this embodiment, the CLIP model is used as the visual encoder, so a fixed size of 336x336 is used. The bounding boxes must include the entire potential abnormal area and cannot exceed the image boundaries. The N bounding boxes can overlap.

[0067] Step 221: According to the N bounding boxes selected in step 213, the corresponding area of ​​the input image is intercepted and copied. It should be noted that the input image at this time should maintain the original size and resolution and has not been processed in any way other than reading in.

[0068] Step 222: After the capture is completed, the input image should be scaled according to the input size required by the visual encoder of the multimodal large model. The original image and the N regions captured in step 221 are input into the visual encoder together to obtain the visual features of the input image and its potential abnormal regions.

[0069] In this preferred embodiment, by using visual experts to predict potential abnormal areas, the local areas intercepted according to the guidance of the visual experts have a higher probability of containing abnormal areas, which can help the multimodal large model focus on possible abnormalities and reduce interference from irrelevant background. In addition, by cropping the original input image, these local areas of potential abnormalities can be input into the model at a higher resolution than the full image, thereby providing more opportunities to discover subtle abnormalities. At the same time, because the visual experts are only responsible for providing possible abnormal areas, their own possible prediction errors will not cause serious misjudgments in the large model. The large model still relies on input images and text instructions to determine whether industrial products have abnormalities.

[0070] See also Figures 4 to 6 , providing a preferred implementation method, the implementation method of using industrial anomaly detection data to construct a multimodal knowledge understanding training task in step 1 includes:

[0071] Step 111: Constructing a multimodal knowledge comprehension training task requires an industrial anomaly detection dataset. This knowledge comprehension training task uses a question-and-answer format, where the question primarily determines whether an industrial product in an image has an anomaly. The large language model generates information about the corresponding product's production process based on the product image or description. The supervised answers are generated in two ways: in steps 112 and 113, depending on whether detailed anomaly description text is provided.

[0072] Step 112: For industrial anomaly detection data that is only labeled with defect types, the multimodal knowledge understanding training task requires the use of a large language model to generate a corresponding checklist based on the defect type and corresponding production process information;

[0073] Step 113: For industrial anomaly detection data with detailed anomaly description text annotations, the large language model can be used to directly generate corresponding thought chain answers based on the anomaly description.

[0074] The main benefit of the knowledge understanding task built based on industrial anomaly detection data is that it helps the multimodal large model align the visual features of the anomaly with the descriptive text of the process that caused the anomaly in the production process information, thereby activating the knowledge of the multimodal large model itself, so that the anomaly can be judged more accurately with the help of the production process information.

[0075] See also Figure 4 , further providing a preferred embodiment, the implementation method of using industrial anomaly detection data to construct a plain text knowledge understanding training task in step 1 includes:

[0076] Step 121: The knowledge comprehension training task for pure text also adopts the form of question-answering task. The main problem is to judge whether there is anomaly in the text description of the image. To build the knowledge training task for pure text, we first need to collect pictures of various common industrial prototypes and generate corresponding picture descriptions through the multimodal large model.

[0077] Step 122: Based on the generated prototype descriptions, the large language model is used to augment the product descriptions. This mainly includes product type augmentation and exception augmentation. Product type augmentation augments the prototype descriptions into different subcategories of the same product to increase data diversity. Exception augmentation augments the product descriptions by adding exception-related descriptions to the exceptions. The augmented product descriptions have a high degree of diversity.

[0078] Step 123: Use the large language model to generate the corresponding production process based on the augmented product description, thereby combining the product description and production process as part of the question in the question-answering task. At the same time, the large language model generates the corresponding answer based on the product description and production process, thus forming a complete plain text knowledge understanding task.

[0079] The pure text knowledge comprehension training task in this preferred embodiment uses the same question-answering format as the multimodal knowledge comprehension training task. However, the pure text knowledge comprehension training focuses on guiding the model to utilize information from the production process. Therefore, the pure text knowledge comprehension training task must cover a variety of industrial products and a variety of abnormal situations, allowing the model to learn production process information for anomalies generated by the same process, and then judge industrial product anomalies based on production process information.

[0080] Principle analysis: By adding an image encoder module based on visual experts and constructing different knowledge understanding training tasks, the present invention applies a large multimodal model to the field of industrial anomaly detection, and has the ability to infer possible anomalies from the production process and apply them to anomaly detection. The guidance of visual experts on potential abnormal areas enables the model to pay more attention to possible abnormal areas, while the greater resolution enables the model to detect minor anomalies. After knowledge understanding training in the field of industrial anomaly detection, the trained large multimodal model can complete the task of industrial anomaly detection, and can achieve better anomaly detection results based on the analysis of production process information and good product features and abnormal feature recognition capabilities.

[0081] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. An industrial image anomaly detection method based on a multimodal large model, characterized by: The multimodal large model includes a large language model and an expert-guided image sampling module. The industrial image anomaly detection method includes the following steps: Step 1: Construct a knowledge comprehension training task based on the industrial anomaly detection dataset. The knowledge comprehension training task includes two types: a multimodal knowledge comprehension task based on the production process and a plain text knowledge comprehension task based on the synthesis of the production process. Step 2: Use the expert-guided image sampling module to predict the anomaly map and extract N regions of interest. These N regions of interest are input into the visual encoder along with the original input image to extract visual features and output a visual encoding that can be recognized by the large language model. Step 3: Build a visual language aligned industrial anomaly detection instruction set; Step 4: Fine-tune the multimodal large model using the knowledge understanding training task and the industrial anomaly detection instruction set; Step 5: Use the trained multimodal model to detect anomalies of industrial products: Visual images of industrial products are input, and an expert-guided image sampling module outputs visual features. These visual features are embedded in additional text prompts and instructions for the production process as a whole sequence and input into a trained multimodal large model. The large model then gives a binary judgment or a detailed anomaly description and analysis based on the text instructions, thereby completing industrial anomaly detection.

2. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: The multimodal knowledge comprehension task based on the production process in step 1 is constructed by using a large language model to generate corresponding thought chains based on samples provided by the industrial anomaly detection dataset. The sample information includes the name and image of the industrial product, the production process and manufacturing process, and any anomalies and defects. There are two thought chain generation schemes based on the abnormal annotation conditions in the sample: Solution 1: For data with detailed anomaly descriptions, the large language model directly generates detailed industrial anomaly detection answers in the form of thought chains. Solution 2: For data with only rough labels of abnormal types, generate a checklist according to the production process as a thinking chain.

3. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1 or 2, characterized in that: The pure text knowledge understanding task based on production process synthesis in step 1 is constructed as follows: the multimodal large model generates product descriptions based on some industrial product prototypes collected on the Internet, and augments their types, details, and existing anomalies, and generates industrial anomaly detection answers in the form of a thought chain based on the product description and the corresponding production process.

4. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: Step 2 The process of extracting visual features by the expert-guided image sampling module includes the following steps: Step 21: The input image is first processed by a visual expert to obtain a spatial anomaly map; Step 22: Mask the anomaly map predicted by the visual expert, leaving only the 10% of pixels with the highest anomaly scores in the map, and use the topological structure analysis algorithm to extract N regions of interest, N ≤ 4; Step 23: determine N bounding boxes based on the N regions of interest in step 22, and intercept and copy corresponding areas of the input image according to the bounding boxes as N potential abnormal areas; Step 24: Input the obtained N potential abnormal regions together with the original input image into the visual encoder to extract features as the final visual features of this image.

5. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 4, characterized in that: During training, when the number N of ROIs extracted from the manually annotated segmentation map is less than 4, M ROIs with random positions and sizes are randomly added as augmentations, and N + M ≤ 4. During testing, the number and positions of ROIs are determined from the anomaly map predicted by visual experts according to the topological structure analysis algorithm.

6. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 4, characterized in that: The topology structure analysis algorithm in step 22 is the OpenCV findContours algorithm.

7. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: The process of step 3 to construct a visual-language aligned industrial anomaly detection instruction set is as follows: based on the manually annotated anomaly descriptions of the industrial anomaly detection dataset, a large language model is used to generate a visual-language aligned industrial anomaly detection instruction set; this instruction set includes two tasks: a judgment task that only determines whether the product is defective and an output description of the product, which includes the product's attributes, the appearance of the anomaly, and the cause of the anomaly.

8. The method for detecting anomalies in industrial images based on a multimodal large model according to claim 1, characterized in that: Step 4 trains the multimodal large model on the knowledge understanding task and the visual-language alignment task according to the standard supervised large model fine-tuning method; the training is completed in the form of multi-task mixed training, in which the visual-language alignment task enables the multimodal large model to align the anomalies and defects of industrial products with the text, and the knowledge understanding task enables the multimodal large model to utilize the industrial knowledge included in the production process and detect and analyze anomalies according to the product production process.

Images