An industrial anomaly detection method based on a multi-modal large model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-04-03
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391603A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of industrial visual inspection, and in particular to an industrial anomaly detection method based on a multimodal large model. Background Technology
[0002] With advancements in industrial quality inspection, the demand for automation and intelligentization is growing. Multimodal large models, as an emerging artificial intelligence technology, can simultaneously process image and text data, possessing powerful feature understanding and generation capabilities. Applying large models to industrial anomaly detection is expected to solve the problem of traditional methods' heavy reliance on massive amounts of labeled data and improve the system's semantic understanding of complex defects.
[0003] However, existing industrial anomaly detection technologies mainly rely on traditional deep convolutional neural networks. These methods are typically based on fully supervised learning and require a large number of balanced defect samples for training. In actual production, rare defect samples are extremely difficult to obtain, resulting in poor model generalization ability. In addition, traditional model outputs are mostly simple classification labels or masks, lacking interpretable descriptions of the judgment criteria, and cannot meet the diagnostic needs of industrial scenarios that require "knowing not only what happened, but also why it happened."
[0004] While general-purpose large models possess strong generalization capabilities, their direct application in precision industrial quality inspection often results in "illusion" phenomena due to a lack of specific industry domain knowledge, and they struggle to handle fine-grained visual defect features. To address this, this application proposes an intelligent industrial anomaly detection method based on a multimodal large model. This method introduces a multimodal joint representation construction module to align fine-grained visual features with high-level semantic features. It utilizes retrieval-enhanced generation technology, employing an industrial domain knowledge base as an external knowledge source to enhance the model's understanding of specific industrial standards. Simultaneously, it introduces a thought chain technique and a self-consistent adjudication mechanism to decompose the complex defect diagnosis task into standardized reasoning paths, ensuring the accuracy and logical consistency of the detection results. Summary of the Invention
[0005] The purpose of this application is to provide an industrial anomaly detection method based on a multimodal large model, which combines multimodal feature fusion, domain knowledge retrieval and structured reasoning mechanism, and has high accuracy, strong generalization ability and interpretability, significantly improving the quality inspection efficiency in complex industrial scenarios.
[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides an industrial anomaly detection method based on a multimodal large model, including: Acquire industrial image data to be inspected; The industrial image data is preprocessed to obtain standardized input image data; A multimodal joint representation construction module is constructed; the multimodal joint representation construction module includes: a visual feature extraction module and a semantic feature generation module; The standardized input image data is input into the multimodal joint representation construction module, the visual features of the image are extracted using the visual feature extraction module, and the corresponding text semantic features are generated using the semantic feature generation module. Align the visual features with the text semantic features to construct a unified multimodal input representation; Based on the semantic features of the text, the retrieval enhancement generation technology is used to retrieve domain knowledge related to the current detection task from a pre-built industrial domain knowledge base. When the retrieved domain knowledge matches the detection task, the domain knowledge and the detection task are used to construct a reasoning context. Based on the unified multimodal input representation and the inference context, a standardized inference path is constructed using the thought chain inference module to perform step-by-step inference analysis on the detection task, including identifying abnormal regions, determining defect types, and assessing defect severity, and integrating the inference results. Based on the integrated reasoning results, determine whether there are any abnormal defects in the industrial image. If no abnormal defects are detected, output a normal judgment result. If an abnormal defect is detected, output the corresponding defect type, defect location and severity information, and generate an interpretable detection report containing the abnormal detection conclusion and the corresponding reasoning process.
[0007] Optionally, the industrial image data is preprocessed to obtain standardized input image data, specifically including: Identify the original resolution and color space of industrial images, perform normalization processing, and map pixel values to a standard normal distribution range by calculating the mean and standard deviation of image channels. Adaptive histogram equalization is used to enhance industrial images, limiting the amplification of contrast and smoothing histogram peaks caused by noise. Perform a center cropping operation based on the preset region of interest parameters to remove irrelevant background at the image edges, and combine this with a data augmentation strategy of random rotation or horizontal flipping to obtain the standardized input image data.
[0008] Optionally, the specific process of extracting visual features from the image using the visual feature extraction module is as follows: The standardized input image data is divided into image blocks and embedded using the ViT input mechanism to obtain an image block embedding sequence. Position codes are added to the image patch embedding sequence to obtain the input sequence; The input sequence is visually encoded using a ViT network to obtain a visual feature sequence. Based on the learnable query vector, Q-Former is used to perform query-based reconstruction of the visual feature sequence to obtain a query-enhanced visual feature sequence. A projection adaptation layer is used to map the query-enhanced visual feature sequence to the large language model input embedding space to obtain a visual prefix feature sequence.
[0009] Optionally, the specific process of generating the corresponding text semantic features using the semantic feature generation module is as follows: Based on the preset task, a prompt template construction method is used to obtain image description prompt information; Based on the image description prompts and standardized images, a large-scale visual language model with multimodal perception and cross-modal alignment capabilities is used to obtain the image description text.
[0010] Optionally, the visual features are aligned with the text semantic features to construct a unified multimodal input representation, as follows: The visual features are queried and encoded using the Q-Former structure, and the visual features are mapped to a feature space consistent with the text semantic features through a projection layer, thereby achieving semantic alignment between visual features and semantic text. The aligned visual feature vectors are combined with the corresponding text semantic features to construct a unified multimodal input representation.
[0011] Optionally, the specific process of retrieving domain knowledge related to the current detection task from a pre-built industrial domain knowledge base is as follows: Based on the aforementioned text semantic features, a pre-set prompt word is used to guide the large language model to generate text queries; Based on the text query, a large language model is used to expand the text semantic features into multiple enhanced query items that include synonyms, industry terms, and contextual associations, forming a query matrix; Based on the query matrix, a preset vector similarity algorithm is used to calculate the similarity between the feature vectors of each sub-document block in the pre-built knowledge base and the feature vectors of each sub-document block. Based on the semantic similarity ranking, N candidate knowledge fragments are obtained. The candidate knowledge fragments are weighted and sorted, and the candidate basic blocks are sorted according to the comprehensive score. The Top-K knowledge fragments with the highest scores are selected as the domain knowledge most relevant to the current detection task.
[0012] Optionally, the specific process of the weighted sorting is as follows: Each enhanced query term was used as an evaluation criterion to perform a secondary relevance score on the candidate basic blocks, resulting in the following score: ;in, This represents the total score. Indicates the scoring weight. Indicates the score of the candidate basic block; Based on the scoring results, the candidate basic blocks are sorted, and the Top-K knowledge fragments with the highest scores are selected as the domain knowledge most relevant to the current detection task.
[0013] Optionally, the specific process of constructing a standardized reasoning path using the thought chain reasoning module is as follows: The system uses a prompting strategy to guide the large model to perform anomaly diagnosis and analysis according to a preset structured reasoning process, which includes an observation phase, an association phase, a synthesis phase, and a conclusion phase. The observation phase is used to extract anomalies in industrial images, the association phase is used to analyze the causes of anomalies by combining retrieved domain knowledge, the synthesis phase is used to make a comprehensive judgment on multi-source evidence, and the conclusion phase generates corresponding anomaly diagnosis results, thereby forming a structured reasoning path.
[0014] Optionally, the specific process for generating an interpretable detection report that includes anomaly detection conclusions and corresponding reasoning processes is as follows: If the reasoning results indicate that the image has abnormal defects, the core elements in the diagnostic data are extracted, and a formatted report is generated according to the template using a large language model. The report includes the judgment conclusion, defect type, defect location, severity assessment, potential fault cause, and a summary of the interpretable reasoning process. The abnormal detection conclusions and related diagnostic information are used to generate test results and output to the system terminal or stored in the test result database.
[0015] Optionally, the step "based on the unified multimodal input representation and the inference context, constructing a standardized inference path using the thought chain inference module, performing step-by-step inference analysis on the detection task, including identifying abnormal regions, determining defect types, assessing defect severity, and integrating the inference results" also includes a self-consistent adjudication mechanism, the specific process of which is as follows: After completing a single structured reasoning, multiple independent reasoning samples are performed on the same input information to generate multiple candidate reasoning paths; The candidate reasoning paths are analyzed for consistency using the referee module, and the diagnostic result with the highest consensus is determined through a majority voting strategy, thereby generating the final anomaly detection conclusion.
[0016] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an industrial anomaly detection method based on a multimodal large model, which has the following beneficial effects: High accuracy and generalization ability: By integrating deep visual features with the semantic reasoning ability of large models and introducing a domain knowledge enhancement mechanism (RAG), the problem of traditional methods relying on a large amount of labeled data is effectively solved, and the generalization ability of the model to rare defects and complex scenarios is improved.
[0017] Strong interpretability: By introducing thought chain technology and self-consistent adjudication mechanism, the complex defect diagnosis task is broken down into standardized reasoning paths, which ensures the accuracy and logical consistency of the test results and can generate interpretable reports containing the reasoning process to meet the diagnostic needs of industrial scenarios.
[0018] Fine-grained feature representation: A multi-level feature joint representation mechanism is constructed to align visual features with semantic features generated by a large model, which improves the ability to represent subtle defects and reduces the impact of environmental noise and lighting changes on the detection results.
[0019] Industrial closed-loop application: It not only outputs test results, but also automatically triggers equipment interconnection protection mechanisms according to the severity of the problem, realizing a complete industrial closed loop from intelligent condition monitoring to predictive maintenance of equipment, effectively avoiding more serious machine damage and downtime losses. .Attached Figure Description To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating an industrial anomaly detection method based on a multimodal large model provided in an embodiment of this application; Figure 2 This is a schematic diagram of an industrial image data preprocessing process provided in an embodiment of this application; Figure 3 A schematic diagram of the multimodal joint representation construction module structure provided in an embodiment of this application. Figure 4 A schematic diagram of a RAG with a multi-stage retrieval and rearrangement strategy provided for an embodiment of this application; Figure 5 A schematic diagram of a standardized reasoning path based on a thought chain provided in an embodiment of this application; Figure 6 This application provides a four-leaf metal component to be tested according to an embodiment of the present application; Figure 7 This is a schematic diagram of an anomaly determination and interpretability report generation process provided in an embodiment of this application. Detailed Implementation
[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0024] In one exemplary embodiment, such as Figure 1 As shown, an industrial anomaly detection method based on a multimodal large model is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes the following steps: Step 101: Acquire the industrial image data to be detected.
[0025] Step 102: Preprocess the industrial image data to obtain standardized input image data.
[0026] The preprocessing process includes image normalization, image enhancement, and image cropping to obtain standardized input image data. The image preprocessing workflow is as follows: Figure 2 As shown; After raw images are acquired on an industrial production line using industrial cameras or sensors, they often contain a lot of environmental noise or non-critical background information due to the complex lighting conditions, mechanical vibrations, or differences in shooting angles. Directly inputting these images into the model may lead to feature extraction errors.
[0027] For example, in tasks such as detecting minute scratches or pinholes on the surface of precision metal parts, the original images may contain interference information such as uneven reflection, excessive brightness, or conveyor belt edges. The system first identifies the original resolution and color space of the image and performs normalization processing. By calculating the mean and standard deviation of the image channels, the pixel values are mapped to the standard normal distribution range, eliminating the differences in data dimensions between different batches of images.
[0028] Meanwhile, recognizing that industrial defects often exhibit "extremely subtle and low-contrast" characteristics, the system utilizes adaptive histogram equalization to enhance the image, limiting contrast amplification and smoothing histogram peaks caused by noise. This effectively suppresses background noise while significantly highlighting potential texture details and edge features. Furthermore, the system performs center cropping based on preset region of interest parameters to remove irrelevant background from image edges and combines data augmentation strategies such as random rotation or horizontal flipping to simulate sample morphology under different poses.
[0029] These standardized images serve as key inputs to the subsequent multimodal joint representation construction module, effectively reducing the interference of ambient lighting changes and background noise on the detection results, and laying a high-quality data foundation for the subsequent visual feature extraction module to accurately capture fine-grained defect features.
[0030] Step 103: Construct a multimodal joint representation construction module; the multimodal joint representation construction module includes: a visual feature extraction module and a semantic feature generation module, as shown in the schematic diagram of the multimodal joint representation construction module structure. Figure 3 As shown.
[0031] Step 104: Input the standardized input image data into the multimodal joint representation construction module, extract the visual features of the image using the visual feature extraction module, and generate the corresponding text semantic features using the semantic feature generation module.
[0032] Step 105: Align the visual features with the text semantic features to construct a unified multimodal input representation.
[0033] In steps 103-105, the system uses the image visual feature extraction module and the semantic feature generation module to simultaneously extract information from the standard image in step 102.
[0034] First, the execution steps of the image visual feature extraction module are as follows: Step 1: Based on the ViT input mechanism, perform image patch segmentation and embedding mapping on the normalized image obtained in step S102 to obtain the image patch embedding sequence. ,in, , Indicates the image height. Indicates the image width. This indicates the side length of the image patch.
[0035] Step 2: Embed the sequence into the image patch obtained in Step 1 By adding positional encoding, the input sequence is obtained. ,in, This represents the position encoding matrix.
[0036] Step 3: Use ViT to process the input sequence obtained in Step 2. Visual encoding is performed to obtain a visual feature sequence. .
[0037] Step 4: Based on learnable query vectors The Q-Former obtained in step 3 is used. Perform query-based reconstruction to obtain query-enhanced visual feature sequences. .
[0038] Step 5: Use a projection adaptation layer to apply the results obtained in Step 4. Mapping to the input embedding space of a large language model yields a visual prefix feature sequence. ;in, Represents the projection matrix. This indicates the bias term.
[0039] Step 6: Output visual prefix feature sequence As direct visual input for subsequent steps.
[0040] The execution steps of the semantic feature generation module are as follows: Step 1: Based on the preset task, use the prompt template construction method to obtain image description prompt information. ; Step 2: Based on the image description prompts in Step 1 Using standardized images, and employing a large external visual language model (a pre-trained multimodal model with hundreds of billions of parameters, such as GPT-4o or Gemini 2.5 Pro) with multimodal perception and cross-modal alignment capabilities, image description text is obtained. ; Step 106: Based on the semantic features of the text, use retrieval enhancement generation technology to retrieve domain knowledge related to the current detection task from a pre-built industrial domain knowledge base. When the retrieved domain knowledge matches the detection task, construct a reasoning context with the domain knowledge and the detection task.
[0041] To mitigate the knowledge illusion problem in the subsequent inference stage when the model deals with specific industrial equipment, a retrieval-enhanced generation (RAG) technique is used to retrieve domain knowledge relevant to the current detection task from a pre-built industrial domain knowledge base. A schematic diagram of the knowledge acquisition process based on retrieval-enhanced generation is shown below. Figure 4 As shown; Step 1: Image description text based on the output of step 104 It uses preset prompts to guide a large language model (pre-trained on a large corpus with 7 bytes or more of parameters; open-source Llama or Vicuna series models or closed-source large models can be used) to generate text queries. ; Step 2: Query based on the text obtained in Step 1 The system uses pre-set prompts to guide a large language model (pre-trained on a large corpus with 7 bytes or more of parameters; open-source Llama or Vicuna models or closed-source models via API calls can be used) to expand the model into multiple enhanced query terms containing synonyms, industry terms, and contextual associations, forming a query matrix. ; Step 3: Initial Search. Based on the query matrix obtained in Step 2. It uses a preset vector similarity algorithm and the feature vectors of each sub-document block in a pre-built knowledge base. Similarity calculations are performed. A higher similarity score indicates that the sub-document block is closer to the current query in the multidimensional semantic space, meaning it has a stronger semantic relevance and is more suitable as a query target. The answer is returned. Based on semantic similarity ranking, N candidate knowledge fragments are obtained.
[0042] To quantitatively assess the semantic relevance between query information and knowledge base content, this embodiment uses cosine similarity as an evaluation metric, and its calculation formula is as follows: ; in, Represents text query items The query feature vector obtained after processing by the text encoding model Indicates the first in the knowledge base Feature vectors of each sub-document block.
[0043] To improve retrieval accuracy, the system employs a "parent-child indexing strategy": vector search is performed at the finest "child block" level to accurately match specific technical parameters or defect descriptions. Once a child block is hit, the system automatically backtracks and loads the complete "parent block" content to which that child block belongs, ensuring the acquisition of contextually complete knowledge. In a specific embodiment, when a thin, elongated dark stripe anomaly is detected in the surface image of the steel plate to be inspected, a query vector is generated based on this anomaly feature and a similarity search is performed in the knowledge base. The hit child block content is: "continuous distribution of striped dark stripes with low grayscale." The system does not directly use these fragmented information that may lack context, but instead finds the corresponding parent block based on its associated parent block identifier, such as "Document 1 - Chapter 3 - Section 2 - Block 2". The parent block specifically includes the morphological features and judgment methods of the scratch, which is more semantically complete and more suitable as a candidate basic block for subsequent anomaly detection result interpretation and defect analysis compared to the child block.
[0044] Step 4: Weighted Ranking and Refinement. The system utilizes the enhanced query terms generated in Step 2. As an "evaluator," a secondary score is given to all candidate basic blocks. The weight of each "evaluator's" score depends on its semantic similarity to the original faulty input, as shown in the formula. And each "evaluator" scores the basic blocks. Depending on its semantic similarity to the content of that basic block, the final score is determined by the total score. To rearrange, the formula is: The top-K basic blocks with the highest scores are selected.
[0045] Step 5: Together with the output of step S104, construct a reasoning context rich in expert experience.
[0046] Step 107: Based on the unified multimodal input representation and the inference context, construct a standardized inference path using the thought chain inference module, perform step-by-step inference analysis on the detection task, including identifying abnormal regions, determining defect types, assessing defect severity, and integrating the inference results.
[0047] This stage uses a self-consistent thought chain to guide the output of steps S104-S107 of the large language model synthesis, performing logical reasoning to generate the diagnostic process and results. A standardized reasoning path is constructed based on the thought chain reasoning module to perform step-by-step reasoning analysis on the detection task and integrate the reasoning results. A schematic diagram of the standardized reasoning path based on the thought chain (CoT) is shown below. Figure 5 As shown; The core of this step lies in transforming the black-box model reasoning into a transparent, structured process. The system guides the large model to follow a thought chain template that mimics the expert testing approach: "Observation—Association—Synthesis—Conclusion." Step 1: Observation Phase. The model (pre-trained on a large corpus, with a parameter count of 7B or more, and open source such as the Llama series and Vicuna series) first identifies and lists all abnormal visual elements in the image based on multimodal features (such as "an irregular scratch exists in the upper left corner, with a length of about 2mm"). This phase only describes the phenomenon and does not perform qualitative analysis.
[0048] Step 2: Association Stage. The model matches the abnormal visual elements obtained in Step 1 with the retrieved domain knowledge. For example, it compares "irregular scratches" with "transportation scratch features" or "processing tool marks" from the input knowledge to analyze the probability of their causes.
[0049] Step 3: Integration Stage. The model integrates multi-source evidence (visual features, knowledge base matching degree, historical probability) and eliminates interfering items. Eliminating interfering items refers to performing consistency checks on the multiple candidate defect categories output in Step 1: the system considers not only the morphological features of the abnormal area but also its location attributes, the surface area it belongs to, the process step, the knowledge base retrieval results, and its historical occurrence probability to comprehensively compare each candidate defect category. If a candidate category is similar in local morphology but inconsistent with its typical occurrence location, formation conditions, or knowledge constraint information, it is determined to be an interfering item and eliminated, thus retaining the defect judgment result that is more consistent with the multi-source evidence. For example, "Although the shape resembles a knife mark, the location is on a non-processed surface, and the knowledge base indicates that this area is susceptible to transportation bumps," thus correcting the judgment direction.
[0050] Step 4: Conclusion Stage. Based on the analysis in Steps 1-3, a final diagnostic result is generated, including a defect description and reasoning process.
[0051] Step 5: To further improve the reliability of reasoning, the system introduces a self-consistency-based adjudication mechanism. For the same detection task, the system triggers M independent thought chain reasoning paths in parallel. That is, it triggers steps 1-4 M times, resulting in M conclusions. The generated M conclusions may differ (e.g., 3 are judged as "scratches," and 1 as "cracks"). The adjudication module performs voting statistics and logical consistency scoring on these reasoning paths, selecting the result with the highest consensus and the most complete reasoning logic as the final output, thereby significantly reducing the false detection rate caused by model "illusions."
[0052] Step 108: Determine whether there are any abnormal defects in the industrial image based on the integrated reasoning results. If no abnormal defects are detected, output a normal judgment result. If an abnormal defect is detected, output the corresponding defect type, defect location and severity information, and generate an interpretable detection report containing the abnormal detection conclusion and the corresponding reasoning process.
[0053] After completing the thought chain reasoning in the previous step, the system receives and parses the comprehensive diagnostic data transmitted in step 108. The system first performs a binary classification anomaly determination on the industrial image to be inspected based on the diagnostic data: if the reasoning result indicates that the image does not have any abnormal defects, the system outputs a normal determination result and allows it to proceed; if the reasoning result indicates that the image has abnormal defects, the system extracts the core elements from the diagnostic data, generates a formatted report according to the template using a large language model, and executes subsequent responses.
[0054] With attachment Figure 6 Taking the four-leaf metal component to be tested as an example, the final output and execution process of step 108 are as follows: In this case, step 107, through the combination of multimodal features and the knowledge base, arrived at the comprehensive conclusion that the component "has large-area structural damage and material loss." In step 108, based on this, the system determines that the component is abnormal and immediately outputs structured anomaly detection results and an interpretability detection report to the system terminal. See [link to relevant documentation]. Figure 7 The specific content includes: [Judgment Conclusion]: Abnormal (NG); [Defect Type]: Severe edge chipping / material peeling; [Defect Location]: The upper left blade and the lower right blade (distributed symmetrically in a central manner);
Severity
[0055] [Reasoning Process]: The system detected that the upper left and lower right stress areas of the component had lost their normal metallic luster, exhibiting large areas of roughness and dark spots, as well as obvious material loss. Comparison with equipment maintenance standards and the historical failure database revealed that these physical characteristics exceeded the scope of normal wear and tear, classifying it as abnormal damage. Verification through a self-consistent adjudication mechanism confirmed that the damage was symmetrical structural failure caused by abnormal overload.
[0056] After the system terminal displays the above report, the system encapsulates the equipment status conclusion, raw image data, and generated diagnostic information into JSON format and stores them uniformly in the factory's equipment asset management system or equipment health database in order to establish a full lifecycle file for the component.
[0057] Meanwhile, in response to such "high-risk" anomalies, the system will automatically trigger the equipment interconnection protection mechanism, sending "emergency pause" or "feed hold" commands to the corresponding CNC system or programmable logic controller, forcing the equipment to stop, and simultaneously pushing "replace parts / tools" maintenance work orders to the mobile terminals of maintenance personnel. This achieves a complete industrial closed loop from intelligent status monitoring to predictive maintenance of equipment, effectively avoiding more serious machine damage and downtime losses.
[0058] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0059] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An industrial anomaly detection method based on a multimodal large model, characterized in that, The industrial anomaly detection method based on a multimodal large model includes: Acquire industrial image data to be inspected; The industrial image data is preprocessed to obtain standardized input image data; A multimodal joint representation construction module is constructed; the multimodal joint representation construction module includes: a visual feature extraction module and a semantic feature generation module; The standardized input image data is input into the multimodal joint representation construction module, the visual features of the image are extracted using the visual feature extraction module, and the corresponding text semantic features are generated using the semantic feature generation module. Align the visual features with the text semantic features to construct a unified multimodal input representation; Based on the semantic features of the text, the retrieval enhancement generation technology is used to retrieve domain knowledge related to the current detection task from a pre-built industrial domain knowledge base. When the retrieved domain knowledge matches the detection task, the domain knowledge and the detection task are used to construct a reasoning context. Based on the unified multimodal input representation and the inference context, a standardized inference path is constructed using the thought chain inference module to perform step-by-step inference analysis on the detection task, including identifying abnormal regions, determining defect types, and assessing defect severity, and integrating the inference results. Based on the integrated reasoning results, determine whether there are any abnormal defects in the industrial image. If no abnormal defects are detected, output a normal judgment result. If an abnormal defect is detected, output the corresponding defect type, defect location and severity information, and generate an interpretable detection report containing the abnormal detection conclusion and the corresponding reasoning process.
2. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The industrial image data is preprocessed to obtain standardized input image data, specifically including: Identify the original resolution and color space of industrial images, perform normalization processing, and map pixel values to a standard normal distribution range by calculating the mean and standard deviation of image channels. Adaptive histogram equalization is used to enhance industrial images, limiting the amplification of contrast and smoothing histogram peaks caused by noise. Perform a center cropping operation based on the preset region of interest parameters to remove irrelevant background at the image edges, and combine this with a data augmentation strategy of random rotation or horizontal flipping to obtain the standardized input image data.
3. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The specific process of extracting visual features from an image using the visual feature extraction module is as follows: The standardized input image data is divided into image blocks and embedded using the ViT input mechanism to obtain an image block embedding sequence. Position codes are added to the image patch embedding sequence to obtain the input sequence; The input sequence is visually encoded using a ViT network to obtain a visual feature sequence. Based on the learnable query vector, Q-Former is used to perform query-based reconstruction of the visual feature sequence to obtain a query-enhanced visual feature sequence. A projection adaptation layer is used to map the query-enhanced visual feature sequence to the large language model input embedding space to obtain a visual prefix feature sequence.
4. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The specific process of generating the corresponding text semantic features using the semantic feature generation module is as follows: Based on the preset task, a prompt template construction method is used to obtain image description prompt information; Based on the image description prompts and standardized images, a large-scale visual language model with multimodal perception and cross-modal alignment capabilities is used to obtain the image description text.
5. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The visual features are aligned with the text semantic features to construct a unified multimodal input representation. The specific process is as follows: The visual features are queried and encoded using the Q-Former structure, and the visual features are mapped to a feature space consistent with the text semantic features through a projection layer, thereby achieving semantic alignment between visual features and semantic text. The aligned visual feature vectors are combined with the corresponding text semantic features to construct a unified multimodal input representation.
6. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The specific process of retrieving domain knowledge related to the current detection task from a pre-built industrial domain knowledge base is as follows: Based on the aforementioned text semantic features, a pre-set prompt word is used to guide the large language model to generate text queries; Based on the text query, a large language model is used to expand the text semantic features into multiple enhanced query items that include synonyms, industry terms, and contextual associations, forming a query matrix; Based on the query matrix, a preset vector similarity algorithm is used to calculate the similarity between the feature vectors of each sub-document block in the pre-built knowledge base and the feature vectors of each sub-document block. Based on the semantic similarity ranking, N candidate knowledge fragments are obtained. The candidate knowledge fragments are weighted and sorted, and the candidate basic blocks are sorted according to the comprehensive score. The Top-K knowledge fragments with the highest scores are selected as the domain knowledge most relevant to the current detection task.
7. The industrial anomaly detection method based on a multimodal large model according to claim 6, characterized in that, The specific process of the weighted sorting is as follows: Each enhanced query term was used as an evaluation criterion to perform a secondary relevance score on the candidate basic blocks, resulting in the following score: ;in, This represents the total score. Indicates the scoring weight. Indicates the score of the candidate basic block; Based on the scoring results, the candidate basic blocks are sorted, and the Top-K knowledge fragments with the highest scores are selected as the domain knowledge most relevant to the current detection task.
8. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The specific process of constructing a standardized reasoning path using the thought chain reasoning module is as follows: The system uses a prompting strategy to guide the large model to perform anomaly diagnosis and analysis according to a preset structured reasoning process, which includes an observation phase, an association phase, a synthesis phase, and a conclusion phase. The observation phase is used to extract anomalies in industrial images, the association phase is used to analyze the causes of anomalies by combining retrieved domain knowledge, the synthesis phase is used to make a comprehensive judgment on multi-source evidence, and the conclusion phase generates corresponding anomaly diagnosis results, thereby forming a structured reasoning path.
9. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The specific process for generating an interpretable detection report that includes anomaly detection conclusions and corresponding reasoning processes is as follows: If the reasoning results indicate that the image has abnormal defects, the core elements in the diagnostic data are extracted, and a formatted report is generated according to the template using a large language model. The report includes the judgment conclusion, defect type, defect location, severity assessment, potential fault cause, and a summary of the interpretable reasoning process. The abnormal detection conclusions and related diagnostic information are used to generate test results and output to the system terminal or stored in the test result database.
10. The industrial anomaly detection method based on a multimodal large model according to claim 1, characterized in that, The step "Based on the unified multimodal input representation and the inference context, construct a standardized inference path using the thought chain inference module, perform step-by-step inference analysis on the detection task, including identifying abnormal regions, determining defect types, assessing defect severity, and integrating the inference results" also includes a self-consistent adjudication mechanism, the specific process of which is as follows: After completing a single structured reasoning, multiple independent reasoning samples are performed on the same input information to generate multiple candidate reasoning paths; The candidate reasoning paths are analyzed for consistency using the referee module, and the diagnostic result with the highest consensus is determined through a majority voting strategy, thereby generating the final anomaly detection conclusion.