Automatic quality control system based on multi-modal large language model

Abstract

The invention relates to the field of automatic control, and particularly discloses an automatic quality control system based on a multi-modal large language model, which comprises the following steps of: firstly, constructing a multi-modal vector knowledge base by using scattered quality documents, CAD (Computer Aided Design) files and standard legend; during detection, the system firstly performs preliminary analysis on a product image to generate a defect hypothesis, and starts a coarse-to-fine two-stage dynamic knowledge retrieval by taking the defect hypothesis as an index. The retrieval process not only matches text semantics, but also finely aligns recalled knowledge with an actual abnormal region in an image through visual positioning, so that a highly focused and credible enhanced context is constructed for a multi-modal large language model. Based on the context, the model can perform deterministic and explainable decision, and finally the general reasoning ability of the model is combined with the harsh standard of a specific industrial scene, so that accurate, flexible and traceable automatic quality control is realized.

Description

Technical Field

[0001] This application relates to the field of automation control, and more specifically, to an automated quality control system based on a multimodal large language model. Background Technology

[0002] In the process of modern industrial manufacturing's deep transformation towards intelligence and automation, product quality control is a core element in ensuring production efficiency and brand reputation. Traditional automated quality control solutions mainly rely on computer vision technology. These solutions identify product defects by deploying deep learning models trained for specific defect types. However, this approach has inherent limitations: First, it heavily relies on large-scale, finely labeled defect datasets. When introducing new product lines or new defects, it requires significant investment in data acquisition and model retraining, lacking flexibility and scalability. Second, traditional computer vision models struggle to understand and execute complex semantic-level quality standards described in natural language, such as "a scratch is unacceptable in area A, but acceptable in area B if its length is less than 2mm," resulting in insufficient capability when handling refined, multi-condition quality inspection tasks.

[0003] With the rise of Multimodal Large Language Models (MLLMs), their powerful vision-language joint understanding and reasoning capabilities offer a new opportunity to solve the aforementioned challenges. Theoretically, MLLMs can directly understand image content and textual quality standards, enabling zero-shot or few-shot defect detection. However, directly applying general-purpose MLLMs to stringent industrial quality inspection scenarios faces a core technical challenge: the impedance mismatch between the general-purpose model and domain-specific knowledge. Specifically, industrial quality inspection knowledge typically exists in explicit, structured forms (such as SOP manuals, CAD drawings, and tolerance specifications), while MLLM knowledge is implicit and static. Furthermore, industrial quality inspection requires deterministic reasoning that strictly adheres to standards, while MLLM's general reasoning is inherently probabilistic, prone to producing illusions that contradict reality, leading to unstable and unreliable judgments. Moreover, limited by the model's context window length, it is not feasible to input a massive quality knowledge base into the model for reasoning all at once.

[0004] Therefore, an optimized automated quality control system is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an automated quality control system based on a multimodal large language model.

[0006] According to one aspect of this application, an automated quality control system based on a multimodal large language model is provided, comprising: The data knowledge base construction module is used to build a multimodal vector knowledge base based on the quality control document set, CAD design file set, and standard legend image set; The product visual defect detection module is used to perform preliminary image analysis and feature extraction on the acquired images of the product to be inspected in order to obtain visual feature vectors and preliminary defect hypotheses. The knowledge context construction module is used to perform two-stage knowledge retrieval and dynamic context construction based on product ID, visual feature vector, and preliminary defect assumptions to obtain enhanced context. The adjudication module is used to make a final adjudication and generate interpretability based on the enhanced context of the product image to obtain a structured adjudication result; The instruction generation module is used to generate robot instructions based on the structured decision results.

[0007] In the aforementioned automated quality control system based on a multimodal large language model, the product visual defect detection module includes: a product multi-scale feature extraction unit, used to extract multi-scale visual features from the image of the product to be inspected to obtain global visual features and local visual feature sets, the global visual features and local visual feature sets forming a visual feature vector; an anomaly localization unit, used to locate the anomalies in the local visual feature sets based on the distribution of normal sample features to obtain anomaly hotspot areas; and a preliminary defect hypothesis generation unit, used to generate preliminary defects hypotheses based on the anomaly hotspot areas in the image of the product to be inspected.

[0008] In the aforementioned automated quality control system based on a multimodal large language model, the product multi-scale feature extraction unit is used to: perform block processing on the image of the product to be inspected to obtain an image block set; and input the image block set into a pre-trained visual encoder to obtain global visual features and local visual feature sets.

[0009] In the aforementioned automated quality control system based on a multimodal large language model, the preliminary defect hypothesis generation unit is used to: visually mark the corresponding area of ​​the product image to be inspected based on the location information recorded in the abnormal hotspot area to obtain a marked image; construct a structured prompt word, wherein the structured prompt word is "Please describe in detail the visual features of the highlighted area in the following figure, and determine whether there are any defects, stains or structural anomalies"; and input the structured prompt word and the marked image into the multimodal large language model to obtain the preliminary defect hypothesis.

[0010] In the aforementioned automated quality control system based on a multimodal large language model, the knowledge context construction module includes: a coarse retrieval unit, used to perform a mixed-modal weighted coarse retrieval based on product ID, visual feature vector, and preliminary defect assumptions to obtain candidate knowledge fragments; a reordering unit, used to perform a refined reordering of the candidate knowledge fragments based on visual positioning to obtain ordered knowledge fragments; and a dynamic structured context assembly unit, used to perform dynamic structured context assembly of the ordered knowledge fragments to obtain enhanced context.

[0011] In the aforementioned automated quality control system based on a multimodal large language model, the coarse retrieval unit includes: a knowledge subset filtering subunit, used to filter knowledge subsets related only to the current product model from the multimodal vector knowledge base using product ID as a metadata filter; a global visual feature extraction subunit, used to extract global visual features from visual feature vectors; a probabilistic embedding subunit, used to probabilistically embed the global visual features and preliminary defect hypotheses to obtain text feature distribution and visual feature distribution; a dynamic weight calculation subunit, used to perform dynamic weight adaptive calculation based on information entropy on the text feature distribution and visual feature distribution to obtain dynamic text weights; and a probability distribution similarity ranking subunit, used to rank the probability distribution similarity of each knowledge fragment in the knowledge subset based on the dynamic text weight, text feature distribution, and visual feature distribution to obtain candidate knowledge fragments.

[0012] In the aforementioned automated quality control system based on a multimodal large language model, the probability distribution similarity ranking subunit is used to: retrieve a first knowledge fragment from a knowledge subset, the first knowledge fragment including a first text vector and a first image vector; calculate the text similarity between the first text vector and the text feature distribution; calculate the visual similarity between the first image vector and the visual feature distribution; and, based on dynamic text weights, perform weighted fusion of the text similarity and visual similarity to obtain a first mixed similarity score.

[0013] In the aforementioned automated quality control system based on a multimodal large language model, the reordering unit is used to: extract a region of interest image from the image of the product to be inspected based on the coordinate information provided by the abnormal hot spot area; input the region of interest image into a visual encoder to obtain a region of interest vector; calculate the positioning score between each knowledge fragment in the candidate knowledge fragment and the region of interest vector; and sort the candidate knowledge fragments in descending order based on the positioning score to obtain the sorted knowledge fragments.

[0014] Compared with existing technologies, this application provides an automated quality control system based on a multimodal large language model, whose... First, a multimodal vector knowledge base is constructed from scattered quality documents, CAD files, and standard drawings. During inspection, the system does not directly make a decision but first performs a preliminary analysis of the product image to generate defect hypotheses. Using these hypotheses as an index, a two-stage dynamic knowledge retrieval process is initiated, progressing from coarse to fine. This retrieval process not only matches text semantics but also uses visual localization to finely align the recalled knowledge with the actual abnormal regions in the image, thereby constructing a highly focused and reliable enhanced context for the multimodal large language model. Based on this context, the model can make deterministic and interpretable decisions. This effectively solves the core technical problems of general models lacking domain knowledge, being prone to illusions in reasoning, and having a limited context window. Ultimately, it combines the model's general reasoning capabilities with the stringent standards of specific industrial scenarios, achieving precise, flexible, and traceable automated quality control. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 This is a system block diagram of an automated quality control system based on a multimodal large language model according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of data flow in an automated quality control system based on a multimodal large language model according to an embodiment of this application.

[0018] Figure 3 This is a block diagram of a product visual defect detection module in an automated quality control system based on a multimodal large language model, according to an embodiment of this application.

[0019] Figure 4 This is a block diagram of the knowledge context construction module in an automated quality control system based on a multimodal large language model according to an embodiment of this application.

[0020] Figure 5 This is a block diagram of a coarse retrieval unit in an automated quality control system based on a multimodal large language model according to an embodiment of this application. Detailed Implementation

[0021] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0022] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0023] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0024] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0025] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0026] This invention aims to address the technical problem of unreliable judgments caused by the contradiction between the universality of general-purpose multimodal large language models and the proprietary and deterministic nature of industrial standards when applied to industrial quality inspection. To address this problem, this application proposes an automated quality control system based on a multimodal large language model. Specifically, firstly, the system offline preprocesses heterogeneous quality knowledge, including SOP documents, CAD drawings, and standard illustrations, and constructs a structured multimodal vector knowledge base as a fact source. In the actual inspection process, the system does not directly allow the model to make open-ended judgments on product images. Instead, it first performs a preliminary scan of the image through an independent visual analysis module to locate abnormal hotspot areas and generate a preliminary, descriptive defect hypothesis. This hypothesis then acts as a smart probe, triggering a two-stage knowledge retrieval process from coarse to fine: In the first stage, combining the textual hypothesis with global image features, a set of candidate quality standards is quickly recalled from the knowledge base; in the crucial second stage, the system further performs precise visual similarity comparison and reordering between the actual abnormal areas in the image and the illustrations attached to the candidate standards. Through this refinement process, the knowledge fragments that are highly matched visually and semantically to the current defects are dynamically assembled into an enhanced context and submitted to the large language model along with the image to be inspected. This process transforms the model's task from blind guessing to adjudication based on solid evidence, thereby greatly suppressing illusions, ensuring the accuracy and interpretability of the output, and ultimately seamlessly converting the structured adjudication results into execution instructions for downstream robots, achieving true knowledge-driven automation quality.

[0027] Figure 1 This is a system block diagram of an automated quality control system based on a multimodal large language model according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow in an automated quality control system based on a multimodal large language model according to an embodiment of this application. Figure 1 and Figure 2As shown, the automated quality control system 100 based on a multimodal large language model according to an embodiment of this application includes: a data knowledge base construction module 110, used to construct a multimodal vector knowledge base based on a quality control document set, a CAD design file set, and a standard legend image set; a product visual defect detection module 120, used to perform preliminary image analysis and feature extraction on the acquired product images to obtain visual feature vectors and preliminary defect hypotheses; a knowledge context construction module 130, used to perform two-stage knowledge retrieval and dynamic context construction based on product ID, visual feature vectors, and preliminary defect hypotheses to obtain enhanced context; an adjudication module 140, used to perform final adjudication and interpretability generation on the product images to obtain structured adjudication results based on the enhanced context; and an instruction generation module 150, used to generate robot instructions based on the structured adjudication results.

[0028] In the aforementioned automated quality control system 100 based on a multimodal large language model, the data knowledge base construction module 110 is used to construct a multimodal vector knowledge base based on a set of quality control documents, a set of CAD design files, and a set of standard illustration images. It should be understood that industrial quality control knowledge is heterogeneously distributed across various data carriers such as textual SOP documents, structured CAD design files, and visual standard illustrations. These original data sources are characterized by their large volume, diverse formats, and unstructured content, making them unsuitable for direct real-time, deterministic understanding and reasoning by the multimodal large language model. Therefore, in the technical solution of this application, a multimodal vector knowledge base is constructed based on the set of quality control documents, the set of CAD design files, and the set of standard illustration images. This preprocesses the dispersed and heterogeneous domain-specific knowledge and transforms it into a structured, computable vector representation that supports efficient retrieval. This provides a stable, reliable, and quickly accessible source of factual evidence for subsequent online quality adjudication, thus laying a solid foundation for the entire system to achieve knowledge-driven decision-making.

[0029] Specifically, in a concrete example of this application, the construction process of this multimodal vector knowledge base involves the systematic analysis and association of the original data sources. First, a document parsing tool is used to extract structured knowledge fragments from a quality control document set. Each fragment contains a quality standard text and its associated product ID metadata. Simultaneously, key views or tolerance annotation areas are extracted from CAD design documents as image information, and qualified and unqualified sample images are obtained from a set of standard legend images. Through preset rules or manual annotation, these extracted text fragments are strongly associated with their corresponding image information, forming multimodal knowledge units containing text, images, and metadata. Subsequently, each multimodal knowledge unit is vectorized. Its text portion is input into a pre-trained text embedding model to generate high-dimensional text vectors; its image portion is fed into a visual encoder to generate high-dimensional image vectors. Finally, the generated text vectors, image vectors, along with their original content and metadata, are stored in a vector database. In this database, efficient retrieval engines are built for text vectors and image vectors respectively, such as an index structure based on the approximate nearest neighbor algorithm, to support high-speed querying and retrieval based on mixed modality similarity in subsequent processes.

[0030] In the aforementioned automated quality control system 100 based on a multimodal large language model, the product visual defect detection module 120 is used to perform preliminary image analysis and feature extraction on the acquired images of the product to be inspected to obtain visual feature vectors and preliminary defect hypotheses. It should be understood that directly using the raw, unstructured images of the product to be inspected in subsequent complex knowledge retrieval and adjudication processes would lead to a lack of clear analytical focus, resulting in vague retrieval targets and wasted computational resources. Therefore, in the technical solution of this application, preliminary image analysis and feature extraction are further performed on the acquired images of the product to be inspected to obtain visual feature vectors and preliminary defect hypotheses, thereby transforming the raw pixel-level information into an intermediate representation containing structured features and textual semantic clues. This provides a precise and efficient query input for the subsequent knowledge retrieval module, thereby greatly improving the targeting and accuracy of the entire quality control process.

[0031] Figure 3 This is a block diagram of a product visual defect detection module in an automated quality control system based on a multimodal large language model, according to an embodiment of this application. Figure 3As shown in the embodiments of this application, the product visual defect detection module 120 includes: a product multi-scale feature extraction unit 121, used to extract multi-scale visual features from the image of the product to be inspected to obtain global visual features and local visual feature sets, the global visual features and local visual feature sets forming a visual feature vector; an anomaly localization unit 122, used to locate the anomaly saliency of the local visual feature set based on the normal sample feature distribution to obtain anomaly hotspot areas; and a preliminary defect hypothesis generation unit 123, used to generate a guided preliminary hypothesis based on the anomaly hotspot areas of the image of the product to be inspected to obtain a preliminary defect hypothesis.

[0032] Specifically, the product multi-scale feature extraction unit 121 is used to extract multi-scale visual features from the product image to obtain global visual features and local visual feature sets, which together form a visual feature vector. It should be understood that single-scale visual analysis cannot simultaneously consider both the overall contextual information of the product and its microscopic local details. Relying solely on global features may overlook minor flaws, while relying solely on local features will lose the ability to determine the correlation between the defect location and the overall product. Therefore, in the technical solution of this application, multi-scale visual feature extraction is further performed on the product image to obtain global visual features and local visual feature sets, which together form a visual feature vector. This synchronously maps the original pixel-level data to a high-dimensional semantic space that can represent both global identity and local details. This provides a complete and hierarchically structured feature foundation for subsequent anomaly localization and knowledge retrieval, thereby ensuring the comprehensiveness and accuracy of the analysis.

[0033] More specifically, in this embodiment, the product multi-scale feature extraction unit is used to: perform block processing on the product image to be inspected to obtain a set of image blocks; and input the set of image blocks into a pre-trained visual encoder to obtain global visual features and local visual feature sets. That is, the multi-scale feature extraction process first performs block processing on the product image to be inspected. The input product image to be inspected is precisely divided into an N×M grid, forming a set of image blocks composed of multiple non-overlapping image blocks. Subsequently, this set of image blocks is input into a pre-trained visual encoder, such as a Vision Transformer model. Inside the encoder, each image block is linearly projected into a feature vector, and together with a classification token specifically used to aggregate global information, it is fed into a multi-layer self-attention network for processing. After processing, the embedding vector corresponding to the classification token is extracted from the output layer of the encoder; this vector is the global visual feature representing the overall content of the image. Simultaneously, the output embedding vectors corresponding to each image block are extracted; these vectors together constitute a local visual feature set, where each local feature vector maintains a clear correspondence with its spatial position in the original image.

[0034] Specifically, the anomaly localization unit 122 is used to locate abnormal hotspots by performing anomaly saliency localization on the local visual feature set based on the distribution of normal sample features. It should be understood that since the massive local visual feature set generated in the previous step is merely a mapping of the original image in the feature space, it does not contain any information about normality or abnormality. Directly performing indiscriminate analysis on all local features would lead to blindness and inefficiency in subsequent processing. Therefore, in the technical solution of this application, anomaly saliency localization is further performed on the local visual feature set based on the distribution of normal sample features to obtain abnormal hotspots, thereby introducing an objective, data-driven benchmark, and using this benchmark to quantitatively evaluate the normality of each local region. In this way, the focus of analysis can be narrowed from the entire image to at least a few regions with a high probability of anomalies, thus achieving rapid, unsupervised preliminary screening and localization of potential defects.

[0035] More specifically, in a concrete example of this application, the anomaly saliency localization process relies on a pre-built normal sample feature distribution model. This model is established offline by inputting images of a large number of qualified products into the same visual encoder, collecting all the generated local visual features, and calculating the mean vector and covariance matrix of these features. During online detection, the system traverses each local feature vector in the local visual feature set of the current product image to be inspected. For each vector, the Mahalanobis distance between it and the pre-stored normal sample feature distribution is used as the anomaly score for that local region. The Mahalanobis distance effectively measures the difference between a data point and a distribution and takes into account the correlation between features. Subsequently, the system compares the calculated anomaly score of each local region with a preset anomaly threshold. All local regions whose anomaly scores exceed the threshold, along with their position coordinates in the original image and their corresponding anomaly scores, are recorded and compiled into a structured data list, which is the anomaly hotspot area, precisely marking the most noteworthy potential anomaly locations on the product image to be inspected.

[0036] Specifically, the preliminary defect hypothesis generation unit 123 is used to generate preliminary defects based on abnormal hotspot areas in the image of the product under inspection. It should be understood that since the abnormal hotspot areas generated in the previous step are merely a set of coordinate data without any semantic information, they cannot serve as effective query conditions for retrieval in a knowledge base primarily composed of natural language. Therefore, in the technical solution of this application, preliminary defects are further generated based on abnormal hotspot areas in the image of the product under inspection. This utilizes the visual understanding and description capabilities of a multimodal large language model to transform the located, unlabeled visual anomalies into a structured, semantically rich natural language text. This generates a textual query strongly related to visual observation, providing an accurate and understandable input for subsequent knowledge retrieval and alignment, effectively bridging the gap between visual anomaly signals and the domain knowledge base.

[0037] More specifically, in this embodiment, the preliminary defect hypothesis generation unit is used to: visually mark the corresponding area of ​​the product image to be inspected based on the location information recorded in the abnormal hotspot area to obtain a marked image; construct a structured prompt, the structured prompt being "Please describe in detail the visual features of the highlighted area in the image below, and determine whether there are any defects, stains, or structural anomalies"; and input the structured prompt and the marked image into a multimodal large language model to obtain the preliminary defect hypothesis. That is, specifically, the generation process of the preliminary defect hypothesis first involves visual marking. Based on the location information recorded in the abnormal hotspot area, the system draws one or more highlighted borders or semi-transparent masks in the corresponding coordinate area on a copy of the product image to be inspected, thereby generating a marked image. This mark explicitly guides the model's attention to the potential abnormal area. Subsequently, the system constructs a structured prompt, which is a preset instructional text, such as "Please describe in detail the visual features of the highlighted area in the image below, and determine whether there are any defects, stains, or structural anomalies." Finally, the marked image containing the visual focus and the structured cue word are submitted as a unified input to a multimodal large language model. After receiving the input, the model follows the instructions of the cue word, specifically analyzes the visual content of the marked region in the image, and generates a descriptive text, which is output as a preliminary defect hypothesis, such as "a fine linear scratch exists in the highlighted area".

[0038] In the aforementioned automated quality control system 100 based on a multimodal large language model, the knowledge context construction module 130 is used to perform a two-stage knowledge retrieval and dynamic context construction based on product ID, visual feature vectors, and preliminary defect hypotheses to obtain enhanced context. It should be understood that, due to the large size and complex content of the multimodal knowledge base, directly using a single-modal or single-stage retrieval method can easily lead to the recall of a large amount of irrelevant or only superficially relevant knowledge due to the semantic ambiguity of the preliminary defect hypotheses or the generality of visual features, failing to ensure a strong correspondence between the retrieval results and specific abnormal regions in the image to be inspected. Therefore, in the technical solution of this application, a two-stage knowledge retrieval and dynamic context construction based on product ID, visual feature vectors, and preliminary defect hypotheses is further performed to obtain enhanced context, thereby implementing a progressive knowledge screening and verification process from broad matching to precise alignment. This ensures that the context information finally provided to the adjudication module is not only semantically relevant but also visually highly consistent with the specific abnormal features detected, thus providing a solid foundation for achieving a high-precision, evidence-based final adjudication.

[0039] Figure 4 This is a block diagram of the knowledge context construction module in an automated quality control system based on a multimodal large language model according to an embodiment of this application. Figure 4As shown in the embodiments of this application, the knowledge context construction module 130 includes: a coarse retrieval unit 131, used to perform a mixed-modal weighted coarse retrieval based on product ID, visual feature vector and preliminary defect hypothesis to obtain candidate knowledge fragments; a re-ranking unit 132, used to perform a fine re-ranking of the candidate knowledge fragments based on visual positioning to obtain ranked knowledge fragments; and a dynamic structured context assembly unit 133, used to perform dynamic structured context assembly on the ranked knowledge fragments to obtain enhanced context.

[0040] Specifically, the coarse retrieval unit 131 is used to perform a mixed-modal weighted coarse retrieval based on product ID, visual feature vector, and preliminary defect hypothesis to obtain candidate knowledge fragments. It should be understood that, since the textual description and visual features of the preliminary defect hypothesis have dynamically changing confidence and information value in different detection scenarios, existing mixed-modal retrieval mechanisms rely on a fixed, preset weighting factor for linear combination when fusing textual and visual information. The inherent rigidity of this method makes it unable to cope with the inherent uncertainties in industrial quality inspection scenarios. Specifically, this mechanism fails to consider the key dynamic element of modal confidence: when the textual description of the preliminary defect hypothesis is highly precise or extremely vague, its information value changes significantly compared to the corresponding visual features, and the static weights cannot be adaptively adjusted, leading to the over-amplification of information from secondary or unreliable modalities, thus contaminating the retrieval results. Simultaneously, this mechanism treats high-dimensional features as fixed vector points, ignoring the probability distribution characteristics of features in the vector space due to input ambiguity, and only using point-to-point cosine similarity for measurement. This essentially loses the uncertainty information of the features and reduces the robustness of the retrieval process. Therefore, in a preferred embodiment of this application, an adaptive modality fusion retrieval mechanism based on Bayesian inference and information entropy is proposed. This mechanism probabilistically models features and dynamically adjusts modality weights using information entropy, ultimately using a distance metric for ranking. Specifically, in the technical solution of this application, a mixed-modality weighted coarse retrieval is further performed based on product ID, visual feature vector, and preliminary defect assumptions to obtain candidate knowledge fragments. This allows for not only pre-filtering using product IDs to narrow the retrieval scope during the retrieval process, but more importantly, it implements an adaptive retrieval strategy that dynamically adjusts the fusion weights based on the uncertainty of each modality input. This ensures recall efficiency while intelligently balancing the guiding role of textual semantic cues and global visual features, effectively suppressing retrieval bias caused by the ambiguity of single-modality information. This provides a controllable and highly relevant set of candidate knowledge fragments for the subsequent refined re-ranking stage, significantly improving the robustness and accuracy of the entire knowledge retrieval process.

[0041] Figure 5This is a block diagram of a coarse retrieval unit in an automated quality control system based on a multimodal large language model according to an embodiment of this application. Figure 5 As shown in the embodiments of this application, the coarse retrieval unit 131 includes: a knowledge subset filtering subunit 1311, used to filter out knowledge subsets related only to the current product model from the multimodal vector knowledge base using product ID as a metadata filter; a global visual feature extraction subunit 1312, used to extract global visual features from the visual feature vector; a probabilistic embedding subunit 1313, used to perform probabilistic embedding on the global visual features and preliminary defect hypothesis to obtain text feature distribution and visual feature distribution; a dynamic weight calculation subunit 1314, used to perform dynamic weight adaptive calculation based on information entropy on the text feature distribution and visual feature distribution to obtain dynamic text weight; and a probability distribution similarity ranking subunit 1315, used to rank the probability distribution similarity of each knowledge fragment in the knowledge subset based on the dynamic text weight, text feature distribution, and visual feature distribution to obtain candidate knowledge fragments.

[0042] Accordingly, the knowledge subset filtering subunit 1311 is used to filter out a subset of knowledge related only to the current product model from the multimodal vector knowledge base using the product ID as a metadata filter. It should be understood that since the multimodal vector knowledge base gathers a massive amount of quality standards covering all product models, and these standards are highly product-specific, performing indiscriminate similarity retrieval across the entire knowledge base would not only incur huge computational resource overhead but also severely interfere with the accuracy of the retrieval results due to the introduction of standards from a large number of irrelevant products. Therefore, in the technical solution of this application, a subset of knowledge related only to the current product model is further filtered out from the multimodal vector knowledge base using the product ID as a metadata filter. This precisely constrains the retrieval space to the knowledge range directly related to the product to be checked before performing the core vector similarity calculation. This fundamentally eliminates cross-product model knowledge interference, greatly improves retrieval efficiency and the accuracy of subsequent recall results, and lays a pure and highly relevant candidate foundation for subsequent mixed-modal retrieval.

[0043] More specifically, in a concrete example of this application, the filtering process is predicated on the knowledge base construction phase, where each knowledge fragment stored in the vector database is appended with its corresponding product ID as metadata tag. When the coarse search process begins, the system first obtains the product ID of the product to be checked. Subsequently, the system sends a query instruction with metadata filtering conditions to the vector database. This instruction requires the database to scan all knowledge entries before performing any vector similarity calculations and to return only those entries whose metadata tags exactly match the current product ID. After the database performs this filtering operation, the output is a significantly reduced and highly relevant subset of knowledge; subsequent mixed-modal similarity ranking will only be performed within this subset.

[0044] Accordingly, the global visual feature extraction subunit 1312 is used to extract global visual features from the visual feature vector. It should be understood that relying solely on the textual description of preliminary defect assumptions for retrieval will, due to its inherent semantic generalization, fail to distinguish between scenarios with vastly different appearances but similar defect descriptions. For example, scratches on a metal casing and scratches on a glass panel may share similar textual descriptions but correspond to completely different quality standards. Therefore, in the technical solution of this application, global visual features are further extracted from the visual feature vector to introduce a macroscopic visual context that can characterize the overall appearance, material, and form of the product into the retrieval process. This effectively binds the semantic defect description with the visual product identity, significantly improving the accuracy of mixed-modal retrieval in distinguishing similar defect types applied to different product backgrounds, thereby ensuring the macroscopic relevance of the recall results in the coarse retrieval stage.

[0045] Accordingly, the probabilistic embedding subunit 1313 is used to probabilistically embed the global visual features and preliminary defect hypotheses to obtain the text feature distribution and visual feature distribution. It should be understood that processing the preliminary defect hypotheses and global visual features into deterministic, single feature vector points ignores the inherent uncertainties of these two information sources in different scenarios. For example, a vague preliminary hypothesis (e.g., the surface appears to have an anomaly) has drastically different information value and reliability compared to a clear hypothesis (e.g., there is a 5mm scratch). A single vector point cannot represent this difference, leading to low-quality information being treated equally with high-quality information in subsequent fusion retrieval, thus reducing the robustness of the retrieval process. Therefore, in the technical solution of this application, instead of using a standard embedding model to generate a single vector for the input preliminary defect hypothesis text and global visual features, probabilistic embedding techniques are used, such as applying Monte Carlo loss at the back end of the embedding model for multiple random forward propagations, thereby generating a set of feature samples for each modality. Subsequently, based on this set of samples, a corresponding multivariate normal distribution is fitted through maximum likelihood estimation, and the text and visual query features are modeled as random variables respectively. (Text feature distribution) and (Visual feature distribution), thereby elevating a single feature point into a probability distribution that can quantify uncertainty. This is expressed by the following formula: in, Represents the distribution of text features. Represents the distribution of visual features. Represents a multivariate normal distribution. Let be the mean vector of the distribution. Let be the covariance matrix. This successfully transforms the input text and visual information from a fixed vector point into a probability distribution containing mean and variance information. In quality inspection scenarios, clear input corresponds to a compact distribution, while ambiguous input corresponds to a diffuse distribution, providing crucial, quantifiable uncertainty information for subsequent adaptive modality weight adjustment and distance measurement.

[0046] Accordingly, the dynamic weight calculation subunit 1314 is used to perform dynamic weight adaptive calculation based on information entropy on the text feature distribution and visual feature distribution to obtain dynamic text weights. It should be understood that although the text and visual feature distributions obtained in the previous step contain uncertain information, if fixed, static fusion weights are used in the subsequent retrieval fusion stage, they cannot adapt to the dynamic changes in the reliability of information from different modalities under different scenarios. For example, a semantically ambiguous preliminary flaw hypothesis and a highly accurate hypothesis will be given equal importance, leading to the over-amplification of information from secondary or unreliable modalities, ultimately contaminating the retrieval results. Therefore, in the technical solution of this application, dynamic weight adaptive calculation based on information entropy is further performed on the text feature distribution and visual feature distribution to obtain dynamic text weights. This utilizes information entropy as a quantitative indicator of the uncertainty of feature distribution and intelligently allocates fusion weights according to the real-time uncertainty of the current input.

[0047] Specifically, in a preferred embodiment of this application, firstly, the information entropy of the text feature distribution and the visual feature distribution are calculated separately, and the information entropy is used as a quantitative indicator of the uncertainty of the feature distribution. For a multivariate normal distribution, its differential entropy... It can be directly calculated from the determinant of its covariance matrix, expressed by the formula: in, It is the differential entropy of the distribution. It is the dimension of the feature space. It is the covariance matrix The determinant of the matrix. Calculate the information entropy of the text feature distributions respectively. Information entropy of visual feature distribution In this mechanism, a higher information entropy value indicates a more diffuse feature distribution, representing greater uncertainty and consequently lower confidence in the modality. Subsequently, a softmax function is applied to the negative entropy (i.e., information content) of the two distributions to adaptively generate dynamic weights for the text modality. This process can be expressed by the following formula: in, These are dynamically calculated text weights, also known as dynamic text weights. It is a temperature coefficient used to adjust the smoothness of weight allocation. This allows for intelligent allocation of fusion weights based on the real-time uncertainty of the current input, ensuring that when the text description is precise, its text feature distribution is compact, and when the text description is precise... Low value When the value approaches 1, the system prioritizes text retrieval; conversely, when visual features are clear but text is blurry, Approaching zero, the system focuses on visual retrieval, thereby achieving dynamic and intelligent control of modal contribution. This ensures that information sources with higher confidence are prioritized to guide retrieval under any circumstances, thus greatly improving the adaptability and final accuracy of hybrid modal retrieval.

[0048] Accordingly, the probability distribution similarity ranking subunit 1315 is used to rank the knowledge fragments in the knowledge subset by probability distribution similarity based on dynamic text weights, text feature distribution, and visual feature distribution to obtain candidate knowledge fragments. It should be understood that traditional cosine similarity measurement methods can only measure the direction between deterministic vectors and cannot comprehensively measure the differences between probability distributions, i.e., they ignore the distribution characteristics of features caused by input ambiguity. If only point-to-point similarity measurement is performed on the mean vector of the feature distribution, the feature uncertainty information carefully modeled in the preceding steps will be lost, thereby reducing the robustness of the retrieval process. Therefore, in the technical solution of this application, the probability distribution similarity ranking of the knowledge fragments in the knowledge subset is further based on dynamic text weights, text feature distribution, and visual feature distribution to obtain candidate knowledge fragments. This adopts an advanced distance metric that can simultaneously perceive the position and uncertainty of features, and combines it with dynamic modal weights for intelligent fusion to achieve a ranking mechanism that is more robust to uncertainty.

[0049] Specifically, in this embodiment of the application, the probability distribution similarity ranking subunit is used to: retrieve a first knowledge fragment from a knowledge subset, the first knowledge fragment including a first text vector and a first image vector; calculate the text similarity between the first text vector and the text feature distribution; calculate the visual similarity between the first image vector and the visual feature distribution; and perform weighted fusion of the text similarity and visual similarity based on dynamic text weights to obtain a first mixed similarity score.

[0050] Specifically, this probability distribution similarity ranking process first traverses every knowledge fragment in the knowledge subset. For any knowledge fragment, its deterministic textual feature vector and visual feature vector stored in the knowledge base are considered as a Dirac delta distribution with a covariance matrix of zero. Subsequently, the system uses the Wasserstein distance, which measures the distance between probability distributions, and its squared formula is as follows: in, and It consists of text feature distribution and visual feature distribution. and It is their mean vector. and It is their covariance matrix. Represents the trace of a matrix. The square of the L2 norm of the vector is used to calculate the distance between the text feature distribution on the query side and the text distribution on the knowledge fragment side, as well as the distance between the visual feature distribution on the query side and the visual distribution on the knowledge fragment side. These two distance values ​​are then converted into corresponding similarity scores. Finally, the system uses the dynamically calculated dynamic text weights from the previous step. The system performs a weighted sum of textual and visual similarity scores to obtain the final mixed similarity score for each knowledge fragment. This process is repeated for all fragments within the knowledge subset. The system then sorts all knowledge fragments in descending order based on the calculated mixed similarity scores and selects the top-ranked fragments as candidate knowledge fragments for final output. This achieves a more robust ranking mechanism to uncertainty, ensuring that the ranking results consider not only the proximity of feature means but also implicitly the uncertainty of the query, making the retrieval results more reliable and accurate, and effectively avoiding retrieval bias caused by uneven information quality.

[0051] In summary, this preferred embodiment can intelligently determine the relative reliability of text and visual modalities based on the specific circumstances of each input detection, dynamically adjusting their influence in the retrieval process. This effectively avoids retrieval bias caused by uneven information quality, significantly improving retrieval accuracy and recall. Secondly, by introducing the quantification and transmission of uncertainty, the robustness of the entire system to fuzzy, noisy, or atypical defect inputs is greatly enhanced. Finally, this mechanism enables the automated quality inspection system based on a multimodal large language model to more flexibly adapt to complex and changing industrial environments, achieving an intelligent upgrade from fixed rules to dynamic decision-making.

[0052] Specifically, the re-ranking unit 132 is used to perform a refined re-ranking of candidate knowledge fragments based on visual localization to obtain ranked knowledge fragments. It should be understood that since the coarse retrieval stage mainly relies on matching global visual features with preliminary defect hypotheses, although the recalled candidate knowledge fragments are related in macro-categories, it does not guarantee that the associated standard legends are precisely consistent with the microscopic visual features of specific abnormal areas in the image of the product to be inspected. This may lead to multiple standards with different appearances but similar defect descriptions being assigned similar initial rankings. Therefore, in the technical solution of this application, the candidate knowledge fragments are further refined based on visual localization to obtain ranked knowledge fragments, thereby introducing a higher-resolution visual comparison step focusing on the abnormal area itself, directly verifying the similarity between candidate knowledge and the actually detected anomaly in local appearance. In this way, knowledge fragments that are not only semantically related but also highly matched with the actual defect in visual details can be accurately elevated to the highest priority, thus providing the most reliable reference for subsequent context construction.

[0053] More specifically, in the embodiments of this application, the reordering unit is used to: extract a region of interest image from the product image to be inspected based on the coordinate information provided by the abnormal hotspot area; input the region of interest image into a visual encoder to obtain a region of interest vector; calculate the positioning score between each knowledge fragment in the candidate knowledge fragments and the region of interest vector; and sort the candidate knowledge fragments in descending order based on the positioning score to obtain the sorted knowledge fragments.

[0054] Specifically, this refined rearrangement process begins with the extraction of the region of interest (ROI) image. Based on the coordinates of anomalous hotspots recorded during the anomaly saliency localization phase, the system precisely crops an image patch containing one or more potential anomalies from the original product image to be inspected; this patch is the ROI image. Subsequently, this ROI image is input into the same visual encoder as in the previous step, and after forward propagation, a compact feature vector is obtained—this is the ROI vector. Next, the system iterates through each knowledge fragment in the candidate knowledge fragment list output from the coarse retrieval phase and extracts the feature vector of its associated standard legend, which is pre-stored in the knowledge base. The system calculates the visual similarity (e.g., cosine similarity) between the standard legend feature vector and the ROI vector generated in the previous step, and uses this similarity value as the localization score for this knowledge fragment. Finally, based on the localization scores calculated for all candidate knowledge fragments, the system performs a descending sort on the entire list, thereby generating a more rationally ordered list of knowledge fragments that has been precisely verified by local visual features.

[0055] Specifically, the dynamic structured context assembly unit 133 is used to dynamically structure the knowledge fragments after sorting to obtain an enhanced context. It should be understood that since the reordered knowledge fragments are still discrete data units, they do not form a logical context that can be directly used by the multimodal large language model for coherent reasoning. Submitting these fragments unordered or in the form of a raw list will increase the difficulty of the model's understanding task and may lead to irrelevant outputs or reasoning biases due to a lack of clear instructions and structure. Therefore, in the technical solution of this application, the sorted knowledge fragments are further dynamically structured to obtain an enhanced context, thereby organically integrating the most critical, multi-round filtered knowledge evidence with the original detection question into a pre-defined, logically clear text framework. This generates a highly focused, information-complete, and formatted final prompt, thereby guiding the multimodal large language model to make decisions according to established logic to the greatest extent possible, ensuring the accuracy, consistency, and interpretability of the final output.

[0056] More specifically, in a concrete example of this application, the dynamic structured context assembly process first performs dynamic selection. The system selects the top-ranked, pre-defined number (e.g., the top three) knowledge fragments from a refined, rearranged list of knowledge fragments as core evidence. Subsequently, the system invokes a predefined text template containing multiple placeholders, such as the product ID to be inspected, preliminary defect hypothesis, relevant standard one: [standard description], relevant standard two: [standard description], etc. Next, the system performs an information filling operation, precisely filling the ID of the current product to be inspected, the previously generated preliminary defect hypothesis text, and the standard text description extracted from the selected knowledge fragments into the corresponding placeholder positions in the template. Finally, this fully filled text string, containing a clear structure and guiding questions, is output as a whole as enhanced context, providing a standardized and readily available input for the final adjudication and instruction generation stages.

[0057] In the aforementioned automated quality control system 100 based on a multimodal large language model, the adjudication module 140 is used to perform final adjudication and interpretability generation on the image of the product to be inspected based on enhanced context to obtain a structured adjudication result. It should be understood that although the enhanced context generated in the previous step provides highly relevant decision-making basis for the multimodal large language model, the model's output is essentially free text. Without constraints, the generated adjudication conclusion may lack a unified format, lack key information, or contain illusory content unrelated to the provided context, making it difficult for the results to be directly parsed and executed by the downstream automated production management system. Therefore, in the technical solution of this application, the final adjudication and interpretability generation on the image of the product to be inspected based on enhanced context is further performed to obtain a structured adjudication result. This constrains the model's generation behavior through a final prompt with a mandatory formatted output instruction, forcing it to perform logical reasoning within the provided knowledge framework and output its conclusions and basis in a deterministic, machine-readable format. In this way, the powerful reasoning capability of the large model can be transformed into stable, reliable, and traceable automated adjudication actions, ensuring the closed loop of the entire quality control process and the industrial applicability of the final output result.

[0058] More specifically, in a concrete example of this application, the final ruling and interpretability generation process begins with constructing a final ruling prompt. The system combines the enhanced context generated in the previous step with an instructional template. This template explicitly instructs the model to make a pass / fail judgment on the situation in the image based on the context content, and to cite specific standards in the context as explanations. Finally, the result is output in JSON format, which includes preset keys such as decision, reasoning, and standard_id. Subsequently, the system submits the image of the product to be inspected along with this final ruling prompt containing enhanced context and formatting instructions to a multimodal large language model. Upon receiving this input, the model comprehensively analyzes the image content and contextual information, and generates a text string conforming to the preset JSON format according to the instructions. Finally, the system parses the JSON string returned by the model, extracts the values ​​corresponding to each key, such as the ruling conclusion, explanatory description, and the standard ID on which it is based, and integrates this information into a structured data object, which is the final structured ruling result.

[0059] In the aforementioned automated quality control system 100 based on a multimodal large language model, the instruction generation module 150 is used to generate robot instructions based on the structured decision results. It should be understood that since the structured decision results output in the previous step are a high-level logical description, they cannot be directly understood or executed by physical execution units such as robots or programmable logic controllers on the production line, creating a semantic gap between automated decision-making and physical actions. Therefore, in the technical solution of this application, robot instructions are further generated based on the structured decision results to translate the abstract, machine-readable decision conclusions into low-level, specific operation sequences that can be executed by specific hardware. This truly connects the entire chain from visual perception to physical execution, achieving a fully automated closed loop in the quality control process and transforming the decision-making capabilities of the intelligent detection system into direct, real-time physical intervention in the production material flow.

[0060] More specifically, in a concrete example of this application, the robot instruction generation process begins with parsing the structured decision result. The system extracts the value of the key decision field from the input JSON object. This value is then fed into a pre-configured instruction mapping module, which maintains a decision-action rule base. This rule base maps different decision results to preset robot action procedures. For example, if the decision field value is "non-conforming," the system triggers the corresponding rule, which points to a robot subroutine for handling non-conforming products. Next, the instruction generation unit generates a sequence of instruction code conforming to the target robot controller language specification (such as RAPID or KRL) based on this subroutine. This code sequence precisely defines the complete motion trajectory and gripper control commands of the robot, from its current position to moving above the product to be inspected, grasping the product, and then moving and placing it in the designated waste recycling area. Finally, the generated instruction code is sent to the robot controller via the industrial Ethernet protocol, which drives the robot to perform the corresponding sorting actions.

[0061] In summary, the automated quality control system based on a multimodal large language model according to the embodiments of this application is explained. It first constructs a multimodal vector knowledge base from scattered quality documents, CAD files, and standard diagrams. During inspection, the system does not directly make a decision, but first performs a preliminary analysis of the product image to generate defect hypotheses. Using these hypotheses as an index, it initiates a two-stage dynamic knowledge retrieval process from coarse to fine. This retrieval process not only matches text semantics but also uses visual positioning to finely align the recalled knowledge with the actual abnormal areas in the image, thereby constructing a highly focused and reliable enhanced context for the multimodal large language model. Based on this context, the model can make deterministic and interpretable decisions. This effectively solves the core technical problems of general models lacking domain knowledge, reasoning prone to illusions, and limited context windows. Ultimately, it combines the model's general reasoning capabilities with the stringent standards of specific industrial scenarios, achieving precise, flexible, and traceable automated quality control.

[0062] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-modal large language model based automated quality control system, characterized in that, The method comprises the following steps: a data knowledge base construction module is used to construct a multi-modal vector knowledge base based on a quality control document set, a CAD design file set and a standard legend picture set; a product visual defect detection module is used to perform image preliminary analysis and feature extraction on the obtained product image to be detected to obtain a visual feature vector and a preliminary defect hypothesis; a knowledge context construction module is used to perform two-stage knowledge retrieval and dynamic context construction based on the product ID, the visual feature vector and the preliminary defect hypothesis to obtain an enhanced context; a decision module is used to perform final decision and interpretability generation on the product image to be detected based on the enhanced context to obtain a structured decision result; an instruction generation module is used to generate robot instructions based on the structured decision result. 2.The multi-modal large language model-based automated quality control system of claim 1, wherein, The product visual defect detection module comprises: a product multi-scale feature extraction unit is used to perform multi-scale visual feature extraction on the product image to be detected to obtain a global visual feature and a local visual feature set, and the global visual feature and the local visual feature set constitute a visual feature vector; an abnormality positioning unit is used to perform abnormal saliency positioning on the local visual feature set based on a normal sample feature distribution to obtain an abnormal hot spot area; a preliminary defect hypothesis generation unit is used to perform guided preliminary hypothesis generation on the product image to be detected based on the abnormal hot spot area to obtain a preliminary defect hypothesis. 3.The multi-modal large language model-based automated quality control system of claim 2, wherein, The product multi-scale feature extraction unit is used to: perform block processing on the product image to be detected to obtain an image block set; input the image block set into a pre-trained visual encoder to obtain the global visual feature and the local visual feature set. 4.The multi-modal large language model-based automated quality control system of claim 2, wherein, The preliminary defect hypothesis generation unit is used to: perform visual marking on the corresponding area of the product image to be detected based on the position information recorded in the abnormal hot spot area to obtain a marked image; construct a structured prompt word, the structured prompt word being "please describe the visual features of the highlighted area in the following figure in detail, and judge whether there is any defect, stain or structural abnormality”; input the structured prompt word and the marked image into a multi-modal large language model to obtain the preliminary defect hypothesis. 5.The multi-modal large language model-based automated quality control system of claim 2, wherein, The knowledge context construction module comprises: a coarse retrieval unit is used to perform hybrid modal weighted coarse retrieval based on the product ID, the visual feature vector and the preliminary defect hypothesis to obtain candidate knowledge segments; a reordering unit is used to perform visual positioning-based fine rearrangement on the candidate knowledge segments to obtain reordered knowledge segments; a dynamic structured context assembly unit is used to perform dynamic structured context assembly on the reordered knowledge segments to obtain an enhanced context. 6.The multi-modal large language model-based automated quality control system of claim 5, wherein, The coarse retrieval unit comprises: a knowledge subset screening subunit is used to filter out a knowledge subset related only to the current product model from the multi-modal vector knowledge base by taking the product ID as a metadata filter; a global visual feature extraction subunit is used to extract a global visual feature from the visual feature vector; a probabilistic embedding subunit is used to perform probabilistic embedding on the global visual feature and the preliminary defect hypothesis to obtain a text feature distribution and a visual feature distribution; a dynamic weight calculation subunit is used to perform dynamic weight adaptive calculation based on information entropy on the text feature distribution and the visual feature distribution to obtain a dynamic text weight; The probability distribution similarity ranking subunit is configured to rank each knowledge segment in the knowledge subset based on the dynamic text weight, the text feature distribution, and the visual feature distribution to obtain the candidate knowledge segments.

7. The automated quality control system based on a multi-modal large language model according to claim 6, wherein, The probability distribution similarity ranking subunit is configured to: obtain a first knowledge segment from the knowledge subset, the first knowledge segment including a first text vector and a first image vector; calculate a text similarity between the first text vector and the text feature distribution; calculate a visual similarity between the first image vector and the visual feature distribution; weight and fuse the text similarity and the visual similarity based on the dynamic text weight to obtain a first hybrid similarity score. 8.The multi-modal large language model-based automated quality control system of claim 5, wherein, The reordering unit is configured to: cut an interest region image from the product image to be inspected according to the coordinate information provided by the abnormal hotspot area; input the interest region image into the visual encoder to obtain an interest region vector; calculate a positioning score between each knowledge segment in the candidate knowledge segments and the interest region vector; rank the candidate knowledge segments in descending order based on the positioning score to obtain the ranked knowledge segments.

Images