Industrial defect information query method based on multi-modal large language model and related device

Abstract

The invention discloses an industrial defect information query method based on a multi-modal large language model and a related device, and the method comprises the steps: inputting an industrial defect image into a visual encoder of a trained multi-modal large language model, and carrying out the extraction to obtain industrial defect visual features; inputting the industrial defect visual features into a cross-modal alignment module of a multi-modal large language model, and extracting to obtain visual representation related to semantics; inputting visual representation related to semantics into an embedding conversion layer, and mapping to generate aligned visual embedding with the same text embedding dimension as the language model; receiving a defect query text input by a user, and converting the defect query text into text embedding; splicing the aligned visual embedding and the text embedding to form a multi-modal joint embedding sequence; and inputting the multi-modal joint embedding sequence into the language model, and outputting industrial defect information corresponding to the defect query text. The invention aims to realize dynamic query of industrial defect information in the manufacturing process.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of industrial manufacturing, and particularly relates to an industrial defect information query method based on a multi-modal large language model and a related device. BACKGROUND

[0002] In the field of industrial manufacturing, to ensure product performance stability, improve manufacturing process efficiency and optimize quality control decisions, accurate identification and root cause analysis of industrial defects generated in the manufacturing process have become a key link. In modern industrial systems, various precision devices (such as semiconductor devices, optical elements, composite material structural parts, etc.) are widely used in high-end equipment, transportation, energy and power, and information infrastructure, and the production efficiency and stability of the manufacturing process are increasingly important, which puts higher requirements on real-time acquisition and intelligent analysis capabilities of industrial defect information in the manufacturing process.

[0003] The manufacturing process of precision devices often needs to go through multiple processing equipment and several complex processing techniques. During this process, industrial defects may be caused by factors such as equipment state drift, process parameter mismatch or environmental interference. Industrial defect identification methods based on deep learning have made remarkable achievements in the field of automated defect identification due to their strong feature extraction capabilities. However, existing methods mostly focus on preliminary classification and detection of defects, and can only provide limited visual or semantic information such as defect quantity and category at a time, lack interactive question and answer capabilities, and cannot meet the in-depth dynamic query requirements of defect information. SUMMARY

[0004] In view of the problems in the prior art, the present application provides an industrial defect information query method based on a multi-modal large language model and a related device, which aims to realize dynamic query of industrial defect information in the manufacturing process, make up for the lack of interactive question and answer capabilities of existing industrial defect identification methods based on deep learning, which can only provide limited visual or semantic information such as defect quantity and category at a time, and cannot meet the in-depth dynamic query requirements, ensure accurate identification of industrial defects and root cause analysis, and meet the higher requirements of real-time acquisition and intelligent analysis of industrial defect information in the manufacturing process.

[0005] To solve the above technical problems, the present application is implemented by the following technical solutions: According to a first aspect of the present application, an industrial defect information query method based on a multi-modal large language model is provided, comprising the following steps: obtaining an industrial defect image to be queried; inputting the industrial defect image into a visual encoder of a trained multi-modal large language model to extract industrial defect visual features; input the industrial defect visual features into a cross-modal alignment module of the multi-modal large language model to extract semantic-related visual representations; input the semantic-related visual representations into an embedding conversion layer to map to obtain aligned visual embeddings with the same text embedding dimension as the language model; receive a defect query text input by a user and convert the defect query text into a text embedding; concatenate the aligned visual embeddings and the text embedding to form a multi-modal joint embedding sequence; input the multi-modal joint embedding sequence into a language model to output industrial defect information corresponding to the defect query text.

[0006] In a possible implementation manner of the first aspect, the visual encoder is a pre-trained visual Transformer model fine-tuned by using a low-rank adaptation technique; the cross-modal alignment module is a pre-trained query Transformer model; the embedding conversion layer is a linear layer fine-tuned by using a low-rank adaptation technique; and the language model is a pre-trained large language model fine-tuned by using a low-rank adaptation technique.

[0007] In a possible implementation manner of the first aspect, the multi-modal large language model is trained by using a two-stage fine-tuning strategy as follows: In a first stage, a visual fine-tuning is performed on the visual encoder by using a prototype supervised contrastive learning framework based on an industrial defect image dataset, to optimize the industrial defect feature extraction capability of the visual encoder; In a second stage, a language fine-tuning is performed on the pre-trained language model based on an industrial defect multi-round dialogue dataset, to inject industrial defect analysis knowledge.

[0008] In a possible implementation manner of the first aspect, the prototype supervised contrastive learning framework adopts a double-branch architecture, including: a classification branch that uses a cross-entropy loss function to drive the visual encoder to learn the class features of the industrial defect images; a contrast branch that uses a supervised contrastive loss function with introduced class prototypes to guide the intra-class aggregation and inter-class separation of the defect features.

[0009] In a possible implementation manner of the first aspect, the total loss function of the visual fine-tuning stage is:

[0010] wherein, Ltotal is the total loss of the visual fine-tuning stage; Lcls is the cross-entropy loss of the classification branch, Lcon is the prototype supervised contrastive loss of the contrast branch, and a hyperparameter for controlling a cross-entropy loss of the classification branch and a prototype supervised contrast loss weight of the contrast branch.

[0011] In a possible implementation manner of the first aspect, the second-stage language fine-tuning adopts a self-recurrent manner, and a loss function is a cross-entropy loss based on the multi-turn dialogue dataset, which is used to measure a difference between a predicted distribution of a language model and a real answer label.

[0012] In a possible implementation manner of the first aspect, the cross-entropy loss function based on the multi-turn dialogue dataset is as follows:

[0013] wherein, represents a cross-entropy loss based on the multi-turn dialogue dataset; represents a length of answer text in the i th turn of dialogue; i represents a first word before the answer text in the i th turn of dialogue; represents a second word before the answer text in the i th turn of dialogue; represents a third word before the answer text in the i th turn of dialogue; represents a fourth word before the answer text in the i th turn of dialogue; represents a fifth word before the answer text in the i th turn of dialogue; represents a sixth word before the answer text in the i th turn of dialogue; represents a seventh word before the answer text in the i th turn of dialogue; represents a predicted probability of represents an aligned visual embedding; represents a text embedding.

[0014] In a possible implementation manner of the first aspect, the industrial defect multi-turn dialogue dataset is constructed according to industrial defect analysis knowledge, and dialogue content revolves around a same defect image, and information of a class, a feature, a root cause and a maintenance measure of the defect is queried in turn.

[0015] According to a second aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the industrial defect information query method based on a multi-modal large language model when executing the computer program.

[0016] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the industrial defect information query method based on a multi-modal large language model.

[0017] Compared with the prior art, the present application has at least the following beneficial effects: ​Existing deep learning-based industrial defect identification methods mostly only target the preliminary classification and detection of defects, providing limited visual or semantic information such as the number and category of defects at a time. They lack interactive question-and-answer capabilities and cannot meet the needs for in-depth and dynamic querying of defect information. In contrast, this invention constructs an end-to-end multimodal large language model for industrial defect information query. It receives the defect query text input by the user, concatenates it with the aligned visual embeddings extracted and transformed from industrial defect images to form a multimodal joint embedding sequence, and then inputs it into a pre-trained language model to output the corresponding industrial defect information. This allows users to dynamically query industrial defect information in a question-and-answer format to obtain more in-depth and comprehensive relevant information. This invention utilizes a visual encoder to extract visual features of industrial defects, and then establishes semantic associations between these features and a pre-trained language model through a cross-modal alignment module. This enables a more accurate understanding of industrial defect image information. Simultaneously, a two-stage fine-tuning strategy based on low-rank adaptation is used to train the model. In the first stage, a visual fine-tuning method based on prototype-supervised contrastive learning is introduced. By introducing category prototypes to guide and improve the intra-class clustering and inter-class separation of defect features, different types of industrial defects can be more accurately identified. In the second stage, a multi-turn dialogue dataset of industrial defects, constructed based on industrial defect analysis knowledge, is used for language fine-tuning. This injects industrial defect analysis knowledge into the model, enabling it to better analyze the root causes of defects. This multi-turn dialogue dataset based on actual industrial defect analysis knowledge allows the model to better adapt to the complex and varied defect query needs in industrial manufacturing scenarios. In practical applications, when faced with user questions about industrial defects raised in different ways and from different perspectives, the model trained on this dataset can understand and process diverse queries. This two-stage fine-tuning strategy optimizes the model from both visual and linguistic levels, comprehensively improving model performance and providing users with accurate and comprehensive industrial defect information.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the specific embodiments of the present invention, the drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an industrial defect information query method based on a multimodal large language model according to the present invention; Figure 2 This invention provides a multimodal large language model framework. Figure 3This describes the specific network structures of the pre-trained visual Transformer, query Transformer, and pre-trained language model with low-rank adaptation fine-tuning in this invention. Figure 4 This is the prototype supervised contrastive learning framework of the present invention; Figure 5 These are the defect types in the wafer image defect dataset used in the embodiments of the present invention; Figure 6 and Figure 7 This is an example of a question-and-answer session between the model described in this embodiment of the invention and three comparative models regarding the same wafer image defect. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Combination Figure 1 and Figure 2 As shown in the figure, this invention provides an industrial defect information query method based on a multimodal large language model. This method utilizes a specially trained multimodal large language model to process industrial defect query tasks. This model can generate a response of industrial defect information based on the input defect image and the user's query text. Specifically, it includes the following steps: S1. Obtain the industrial defect image to be queried.

[0023] In this step, industrial defect images can be acquired using image acquisition devices such as industrial cameras, microscope cameras, or scanning electron microscopes (SEMs) deployed on manufacturing equipment. For example, during semiconductor chip manufacturing, images are taken of the wafer surface to obtain images of abnormal areas that may include scratches, particle contamination, lithographic defects, etc. It should be understood that the images can be acquired in real time or retrieved from a database storing historical defect data.

[0024] Preferably, in order to meet the input requirements of the multimodal large language model visual encoder, all industrial defect images are converted into RGB images with a pixel size of 224×224.

[0025] S2. Input the industrial defect image into the visual encoder of the trained multimodal large language model to extract the visual features of the industrial defects.

[0026] Specifically, the visual encoder is responsible for extracting deep features from the input industrial defect image.

[0027] In a preferred implementation, the visual encoder is a pre-trained visual Transformer model fine-tuned using low-rank adaptation. Low-rank adaptation adapts the model to downstream tasks by injecting a trainable low-rank decomposition matrix into the original model structure. This efficiently utilizes the powerful feature extraction capabilities of the pre-trained model while significantly reducing the computational resources and number of parameters required for fine-tuning, allowing it to focus on learning the unique visual patterns of industrial defects.

[0028] S3. Input the visual features of the industrial defects into the cross-modal alignment module of the multimodal large language model to extract the semantically related visual representation.

[0029] Specifically, the cross-modal alignment module is used to establish a bridge between visual features and the semantic space. The cross-modal alignment module transforms the high-dimensional, abstract visual features extracted in step S2, enabling the visual features to be understood and utilized by the language model.

[0030] In one possible implementation, the cross-modal alignment module is a pre-trained query Transformer model that reconstructs visual features through an attention mechanism to generate an intermediate representation that closely resembles the semantic structure of the text.

[0031] S4. Input the semantically related visual representation into the embedding transformation layer to generate an aligned visual embedding with the same text embedding dimension as the language model.

[0032] Specifically, the embedding transformation layer typically consists of one or more linear layers (fully connected layers). Its function is to map the dimensions of the visual representation output by the cross-modal alignment module to a space that is completely consistent with the text word embedding dimensions of the pre-trained language model.

[0033] In one possible implementation, the linear layer also employs low-rank adaptation techniques for fine-tuning to improve the efficiency of the conversion process.

[0034] S5. Receive the defect query text input by the user and convert the defect query text into text embedding.

[0035] Specifically, users input query text in natural language, such as: "What type of defect is there in this image?", "What are the possible causes of this defect?", or "How can this defect be prevented?". After receiving the text, the system uses the lexicalization and embedding layers of the pre-trained language model to convert the input text into a series of text embedding vectors.

[0036] S6. The aligned visual embedding and the text embedding are concatenated to form a multimodal joint embedding sequence.

[0037] This step merges the aligned visual embeddings with the text embeddings. Specifically, the aligned visual embedding is treated as a special "visual symbol," and it is concatenated with the text embedding sequence along the sequence length dimension to form a multimodal joint embedding sequence, which serves as the input to the pre-trained language model.

[0038] S7. Input the multimodal joint embedding sequence into the language model and output the industrial defect information corresponding to the defect query text.

[0039] The language model (e.g., ChatGLM-6B) receives the aforementioned multimodal joint embedding sequence. Since the language model has been pre-trained on massive amounts of text, it possesses powerful language understanding and generation capabilities. Furthermore, in this scheme, it has been fine-tuned using low-rank adaptation techniques, enabling it to understand the defect information represented by visual embeddings and the user intent represented by text embeddings in the sequence. The model generates a response word sequence in an autoregressive manner, ultimately outputting industrial defect information in natural language form to complete the query.

[0040] In one possible implementation, the multimodal large language model is trained through a first-stage and second-stage fine-tuning strategy that effectively combines industrial defect visual feature learning with defect analysis knowledge injection.

[0041] Specifically, the first stage involves fine-tuning the visual encoder. Based on an industrial defect image dataset, this stage employs a prototype-supervised contrastive learning framework to fine-tune the visual encoder, optimizing its feature extraction capabilities for industrial defects. This framework uses a dual-branch architecture to optimize two objectives in parallel. The dual-branch architecture includes a classification branch and a contrastive branch. The classification branch uses a cross-entropy loss function to drive the visual encoder to learn to distinguish different categories of industrial defects, enhancing its category feature extraction capabilities. The contrastive branch uses a supervised contrastive loss function that incorporates category prototypes. This loss function not only narrows the distance between features within the same category and widens the distance between features from different categories, but also, by introducing learnable category prototype vectors as anchors, more effectively guides defect features to cluster within categories and separate between categories in the feature space.

[0042] The overall loss function for the visual fine-tuning stage is:

[0043] in, This represents the overall loss during the visual fine-tuning phase. The cross-entropy loss of the classification branch, The prototype-supervised contrastive loss for the contrastive branch is... and The hyperparameters are used to control the cross-entropy loss of the classification branch and the prototype-supervised contrastive loss weights of the contrastive branch.

[0044] The second stage involves fine-tuning the language model. Based on an industrial defect multi-turn dialogue dataset, this stage fine-tunes the language model (e.g., ChatGLM-6B) to infuse it with professional industrial defect analysis knowledge, enabling it to answer questions in relevant domains. This industrial defect multi-turn dialogue dataset is constructed based on industrial defect analysis knowledge (such as process manuals, equipment maintenance logs, and expert experience). The dialogue content revolves around the same defect image, simulating a real human-machine interaction scenario, sequentially querying information such as the defect category, appearance characteristics, root cause, and suggested maintenance measures. This stage employs an autoregressive approach for fine-tuning, with the loss function being the cross-entropy loss based on the multi-turn dialogue dataset, used to measure the difference between the language model's predicted symbol distribution and the actual answer labels. The specific expression of the loss function is as follows:

[0045] In the formula, This represents the cross-entropy loss based on the multi-turn dialogue dataset; Indicates the first i The length of the response text in a turn-based dialogue; Indicates the first Before answering the text in a turn-based dialogue One word; Indicates the first Answer the text in the round-robin dialogue One word; express The predicted probability; Indicates alignment with visual embedding; This indicates text embedding.

[0046] Through the above two-stage training, the multimodal large language model of the present invention is finally obtained, which has both the ability to extract visual features of industrial defects and the ability to generate language for defect analysis, and can effectively realize dynamic interactive query of industrial defect information.

[0047] In detail, to avoid the high demands of full fine-tuning on large-scale computing resources and time, this implementation adopts a low-rank adaptation fine-tuning method. Multiple adjustable pairwise decomposition matrices are inserted into the architecture of the pre-trained visual Transformer and the pre-trained language model, and parameters are updated, while the original pre-training parameters are frozen. Since the pre-trained visual Transformer lacks feature extraction capabilities for industrial defect images, and the pre-trained language model lacks industrial defect analysis knowledge, this invention adopts a two-stage fine-tuning strategy based on low-rank adaptation: first, visual fine-tuning optimizes the industrial defect feature extraction capability of the pre-trained visual Transformer; then, language fine-tuning injects industrial defect analysis knowledge into the pre-trained language model.

[0048] like Figure 2 As shown, the multimodal large language model for industrial defect information query consists of a low-rank adapted pre-trained visual Transformer, a query Transformer, a low-rank adapted fine-tuned embedding transformation layer, and a low-rank adapted fine-tuned pre-trained language model. First, the industrial defect image is segmented into image patches, passed through a linear projection layer, and then input into the visual Transformer. The pre-trained visual Transformer uses a pre-trained CLIP model as a teacher model, performs self-training through a masked image modeling task, and finally expands its parameters to a larger scale. The visual Transformer acts as a visual encoder to extract features from the industrial defect image. Figure 3 As shown in (a), the specific structure of the Visual Transformer consists of alternating stacked multi-head self-attention layers and feedforward layers (a multilayer perceptron with a single hidden layer), with residual connections and layer normalization operations applied to each layer. The image features extracted by the Visual Transformer are finally mapped to feature vectors through linear layers. The specific structure of the cross-modal alignment Transformer is shown in [reference needed]. Figure 3 As shown in (b), it consists of stacked multi-head self-attention layers, cross-attention layers, and feedforward layers. The pre-trained query first interacts with each other through the multi-head self-attention layers, and then interacts with the image features output by the visual Transformer through the cross-attention layer, extracting text-related visual representations from the visual features. Then, the visual representations are mapped to the same dimensional space as the text embeddings of the language model through an embedding transformation layer (linear layer), generating aligned visual embeddings, thus achieving modal transformation from vision to language. The language model has the same specific structure as the visual Transformer, and it is pre-trained on a large-scale text dataset. Before adding the mapped visual embeddings to the input text embeddings, the language model receives the merged visual and text embeddings and outputs the corresponding response accordingly.

[0049] like Figure 3As shown, based on the idea that the incremental parameter matrix obtained from large model updates contains a large number of redundant parameters, low-rank adaptation is used to perform low-rank decomposition approximation on the incremental parameter matrix. Specifically, for the pre-trained weight matrix... Its incremental parameter matrix Represented as and ,and Therefore, the updated large model parameter matrix The fine-tuning process is simplified to training two decomposition matrices with relatively small parameter sizes.

[0050] Prototype-supervised contrastive learning framework, such as Figure 4 As shown. First, the industrial defect image dataset... Image samples Three different random augmentations were performed, resulting in three different augmented datasets. , and . Used for classification branches, it can be regarded as a common random augmentation for classification tasks. and For comparison branches, samples of the same class in two datasets are considered positive samples of each other.

[0051] The contrastive branch extracts features using a visual Transformer and a non-linear projection head. To enhance intra-class clustering in feature extraction guided by category information, a category prototype derived from the classifier's weight matrix is ​​introduced. , Indicates the first p The class's category prototype. Specifically, the classifier. weight matrix It can be represented as Image feature vectors and Cosine similarity measures whether an image belongs to the first cosine similarity. p The confidence level of the class. Therefore, It can be regarded as the category center of the category in the feature space. The result is obtained through a multilayer perceptron with a single hidden layer. , see the following formula for details: (1) Introduce the category prototype in each batch, and let The prototype-supervised contrastive loss is (note: the set in the formula is temporarily considered as the set of its element indices): (2) In the formula, express The feature vector extracted by the visual Transformer express The middle belongs to the tag a subset of samples This represents the temperature coefficient.

[0052] Furthermore, the influence of the majority class is reduced by averaging the gradient contribution of the similarity index of each class to the loss function, thus obtaining the final prototype-supervised contrastive loss: (3) Note: When , Become ,right Similarly.

[0053] In addition, the cross-entropy loss function is used to train the classification branch: (4) In summary, the loss function for visual fine-tuning is: (5) The query Transformer transforms the industrial defect features extracted by the visual Transformer after visual fine-tuning into text-related visual representations, and then maps them to visual embeddings through an embedding transformation layer. Question and answer text of multi-turn dialogue and It was also converted into a text embedding. and abbreviated as Converted to The input sequence is composed of visual embeddings, historical dialogues, and the latest question, with the latest answer serving as the label. For the first round of dialogue, the input is... ; tag is For the first i A round-robin dialogue, the input of which is ; tag is Guided by labels, the language model uses an autoregressive approach to predict the next element of the input sequence word by word in a unidirectional manner, and measures the difference between the predicted distribution and the true label using cross-entropy loss. For a given input sequence with labels... The specific loss formula for the defective image of turn-based dialogue is as follows: (6) The present invention will be further described in detail below with reference to specific embodiments, which specifically include the following steps: Step 1: Perform electrical performance tests on wafers collected from a real wafer fabrication plant to obtain a mixed defect dataset of wafer images. Each wafer image in this dataset is 52×52 pixels, and each element on the wafer image has only three possible values: 0, 1, and 2. 0 represents a blank point (no die at this location), 1 represents a normal die that passed the electrical test, and 2 represents a failed die that failed the electrical test. Figure 5 As shown, this dataset includes a normal type and eight single defect types: Center defect (C), Donut defect (D), Edge_Loc defect (EL), Edge_Ring defect (ER), Local defect (Loc, L), Near_Full defect (NF), Scratch defect (S), and Random defect (R). By combining these single defects, 13 dual-mixed defect types, 12 triple-mixed defect types, and 4 quadruple-mixed defect types (e.g., C+L+EL+S includes four defect types: C, L, EL, and S) are generated, covering a total of 38 defect types. The sample size for each defect type is detailed in Table 1.

[0054] Table 1. Distribution of sample size for each defect type

[0055] Step 2: To adapt to the input requirements of the multimodal large language model visual encoder, all wafer images were converted into 224×224 pixel RGB images. For C24 type defects, the first 800 samples were assigned to the training set, and the last 200 samples were assigned to the test set; for other defect types, the first 80% of the samples were assigned to the training set, and the last 20% of the samples were assigned to the test set.

[0056] Step 3: Construct a multimodal large language model. The specific architecture parameters and low-rank adaptation configuration details are shown in Table 2.

[0057] The specific architecture parameters and low-rank adaptation configuration details of the models described in Table 2

[0058] Step 4: Construct a multi-turn dialogue dataset based on domain-specific analytical knowledge of industrial defects. Specifically, set up dialogue content that sequentially queries four types of information: defect category, characteristics, root cause, and maintenance measures. For each type of information, set up multiple query methods (i.e., multiple question texts), and respond to the queries based on defect analysis knowledge. Simultaneously, set up a small portion of the dialogues to generate new questions based on the specific responses, thereby enriching the dataset's diversity.

[0059] Step 5: Train the constructed model using a two-stage fine-tuning strategy based on low-rank adaptation. The visual fine-tuning stage performs 60,000 iterations, and the language fine-tuning stage performs 15,000 iterations. The base learning rate is set to 0.001, including a warm-up stage with a ratio of 0.02, and the batch size is set to 10. In the first stage, a two-branch framework based on prototype-supervised contrastive learning is used to optimize the defect feature extraction capability of the visual Transformer. Specifically, for the classification branch... Data augmentation is performed using random rotation and random grayscale transformation, compared with branching. and Random flipping and random color dithering are employed; the weight hyperparameter in the prototype-supervised contrastive loss is also used. and The temperature coefficients were set to 1.0 and 0.35 respectively. The dimensions of the hidden and output layers of the multilayer perceptron used for nonlinear projection and classifier weight transformation are set to 1408 and 256, respectively. In the second stage, the low-rank adaptation matrix of the final layer of the visual Transformer is updated to optimize cross-modal alignment. In the multi-turn dialogue dataset, dialogue turns for normal defect types are set to 2 turns (due to their limited information), while those for other defect types are set to 4 turns.

[0060] Step 6: Support dynamic multi-round queries by users during the inference phase, and finally output the diagnostic results.

[0061] To quantify the performance of this invention in defect identification tasks, four evaluation metrics from classic classification tasks are used: accuracy, precision, recall, and F1 score. The formulas for each metric are as follows: (7) (8) (9) (10) Where TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively.

[0062] To verify the effectiveness of the two-stage fine-tuning strategy and prototype-supervised contrastive learning, ablation experiments were conducted. Under the premise of uniformly preserving language fine-tuning, six ablation scenarios were set up: (1) complete removal of visual fine-tuning (i.e., the visual Transformer is not configured with low-rank adaptation, and only language fine-tuning is used); (2) the visual Transformer is configured with low-rank adaptation but the visual fine-tuning loss function is removed, and the model parameters are jointly optimized only through a single stage using language loss; (3) the visual Transformer is configured with low-rank adaptation but only cross-entropy loss is used to optimize parameters in visual fine-tuning; (4) the visual Transformer is configured with low-rank adaptation and cross-entropy loss and ordinary contrastive loss (InfoNCE loss) are used to optimize parameters in visual fine-tuning; (5) the visual Transformer is configured with low-rank adaptation and cross-entropy loss and supervised contrastive loss (removing class prototypes) are used to optimize parameters in visual fine-tuning; (6) the method presented in this paper is fully implemented. All scenarios used the same implementation details to ensure comparability.

[0063] The recognition accuracy of the present invention in the above ablation scenarios is shown in Table 3. The results show that: (1) consistent with the previous analysis, the pre-trained multimodal large language model has a very low defect recognition accuracy (9.78%) when only language fine-tuning is performed. This is due to the lack of defect feature extraction capability. (2) When the visual Transformer is optimized by only language fine-tuning loss, the recognition accuracy is significantly improved to 88.28%, which is attributed to the effective supervision provided by the category information embedded in the language label. (3) After adopting a two-stage fine-tuning strategy and using cross-entropy loss to optimize the visual Transformer parameters in the first stage, the accuracy is further improved to 97.41%, which is attributed to the fact that the classification label provides more direct supervision for optimizing the category feature extraction capability of the visual Transformer than the language label. (4) On the basis of retaining cross-entropy loss, both adding InfoNCE loss and supervised contrastive loss improve the accuracy. This is because the contrastive loss further optimizes the feature space. The supervised contrast loss introduces label information compared to the InfoNCE loss, thus achieving a higher accuracy improvement; (5) the category prototype improves the intra-class clustering and inter-class separation of defect features, which further improves the accuracy by 0.43%.

[0064] Table 3. Recognition accuracy of the present invention in six ablation scenarios.

[0065] Furthermore, the recognition performance of Visual Transformer, Swin-Transformer, CWDRNet, WM-PeleeNet, and DefectGLM was compared with that of the proposed model. Visual Transformer and Swin-Transformer are computer vision models based on the Transformer architecture; CWDRNet and WM-PeleeNet are state-of-the-art models based on convolutional neural network architectures specifically designed for wafer image defect recognition tasks; DefectGLM is a state-of-the-art visual language model applied to wafer image defect recognition and information retrieval. The evaluation metrics were the average of precision, recall, and F1 score for all categories, as well as the overall accuracy. To eliminate randomness, each model experiment was repeated five times, and the average of the metrics was calculated. Specific results are shown in Table 4.

[0066] Table 4. Comparative experimental results of this invention with other models.

[0067] The experimental results are as follows: Compared with other models, the proposed method achieves the best performance in all four metrics (accuracy 98.59%, precision 98.34%, recall 98.45%, and F1 score 98.39%). Consistent with the previous analysis, the recognition performance of CNN-based WM-PeleeNet and CWDRNet is superior to Visual Transformer, Swin-Transformer, and DefectGLM. Notably, under the proposed visual fine-tuning method, the model outperforms WM-PeleeNet and CWDRNet, with accuracy improvements of 0.56% and 0.40%, and F1 scores improvements of 0.44% and 0.31%, respectively. This result further validates the superiority of the proposed visual fine-tuning method.

[0068] Further qualitative evaluation of the model's ability to respond to defect information queries was conducted. In this evaluation, a single-turn dialogue subset was constructed, consisting of the first-turn dialogues selected from the multi-turn dialogue dataset, to train the model (as the baseline model for the ablation experiment) and the comparison model Qwen2-VL-7B-Instruct (Qwen2-VL). DefectGLM was also selected as a comparison baseline. The model trained on the complete multi-turn dialogue dataset is denoted as the model (multi-turn). Figure 6 and Figure 7 Examples of queries and responses for DefectGLM, Qwen-VL, the single-round model, and the multi-round model on the same query wafer plot are presented. The selected wafer plot with the C+L defect type effectively highlights the main limitations of the comparative benchmark model. Observations are as follows: Figure 6As shown in (a), DefectGLM has poor instruction compliance and often fails to directly respond to user queries. Its output exhibits severe overfitting, resulting in ambiguous responses. For example... Figure 6 As shown in (b), Qwen2-VL occasionally fails to recognize the intent of the query, resulting in irrelevant responses. Figure 7 As shown in (c), the model (single round) is capable of generating high-quality responses. However, compared to the correct responses of the model (multi-round), it occasionally gives incorrect responses, particularly in root cause analysis and maintenance measures. In contrast, as Figure 7 As shown in (d) in the figure, the model (multi-round) is able to accurately understand all questions and provide correct, detailed and high-quality responses.

[0069] This invention constructs a multimodal large language model integrating visual and language understanding, supporting users to query defect information in multiple rounds, dynamically, and in-depth using natural language. This overcomes the limitations of existing methods that only provide one-time classification and detection results, enabling coherent and accurate responses to multi-level information such as defect categories, features, root causes, and maintenance measures. By introducing a visual fine-tuning strategy based on prototype-supervised contrastive learning, the visual encoder effectively enhances the intra-class clustering and inter-class separation of industrial defect features, improving the model's recognition performance in complex industrial scenarios such as inter-class similarity and class imbalance. Experiments show that it outperforms existing mainstream models on multiple evaluation metrics. By constructing a multi-turn dialogue dataset of industrial defects and injecting it into a pre-trained language model during the language fine-tuning stage, the model acquires professional knowledge understanding and reasoning capabilities in the industrial manufacturing field, providing professional responses that meet actual production needs. Employing low-rank adaptation fine-tuning technology, only a small number of parameters are updated, reducing computational resource and training time requirements, while effectively mitigating overfitting caused by the small scale of industrial defect data, thus improving the model's generalization ability. The model can provide coherent responses by incorporating historical dialogue context, allowing users to gradually delve deeper from superficial information queries to root cause analysis and maintenance recommendations, thus enhancing the interactive experience of the system in actual industrial operations and maintenance. The method proposed in this invention does not rely on specific types of industrial defects or equipment and can be widely applied to various precision manufacturing scenarios such as semiconductors, optical components, and composite materials.

[0070] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to operate an industrial defect information query method based on a multimodal large language model.

[0071] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the industrial defect information query method based on a multimodal large language model in the above embodiments.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0076] This invention also provides a computer program product, which is used to execute any of the above-described industrial defect information query methods based on a multimodal large language model. Since the computer program product provided by this invention and the above-described industrial defect information query method based on a multimodal large language model belong to the same inventive concept, the computer program product provided by this invention possesses all the advantages of the above-described industrial defect information query method based on a multimodal large language model. Therefore, the beneficial effects of the computer program product provided by this invention will not be elaborated upon here.

[0077] In this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0078] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A method for querying industrial defect information based on a multi-modal large language model, characterized in that, The method comprises the following steps: acquiring an industrial defect image to be queried; inputting the industrial defect image into a visual encoder of a trained multi-modal large language model to extract industrial defect visual features; inputting the industrial defect visual features into a cross-modal alignment module of the multi-modal large language model to extract semantic-related visual representations; inputting the semantic-related visual representations into an embedding conversion layer to map and generate aligned visual embeddings with the same text embedding dimension as the language model; receiving a defect query text input by a user and converting the defect query text into a text embedding; concatenating the aligned visual embeddings and the text embedding to form a multi-modal joint embedding sequence; inputting the multi-modal joint embedding sequence into a language model to output industrial defect information corresponding to the defect query text.

2. The industrial defect information query method based on a multi-modal large language model according to claim 1, characterized in that, The visual encoder is a pre-trained visual Transformer model fine-tuned by using a low-rank adaptation technique; the cross-modal alignment module is a pre-trained query Transformer model; the embedding conversion layer is a linear layer fine-tuned by using a low-rank adaptation technique; and the language model is a pre-trained large language model fine-tuned by using a low-rank adaptation technique.

3. The industrial defect information query method based on a multi-modal large language model according to claim 1 or 2, characterized in that, The multi-modal large language model is trained by using a two-stage fine-tuning strategy as follows: In a first stage, a visual fine-tuning is performed on the visual encoder by using a prototype supervised contrastive learning framework based on an industrial defect image dataset to optimize the industrial defect feature extraction capability of the visual encoder; In a second stage, a language fine-tuning is performed on the pre-trained language model based on an industrial defect multi-turn dialogue dataset to inject industrial defect analysis knowledge.

4. The industrial defect information query method based on a multi-modal large language model according to claim 3, characterized in that, The prototype supervised contrastive learning framework adopts a double-branch architecture, which comprises: a classification branch that uses a cross-entropy loss function to drive the visual encoder to learn the category features of the industrial defect image; a contrast branch that uses a supervised contrastive loss function with introduced category prototypes to guide the intra-class aggregation and inter-class separation of the defect features.

5. The industrial defect information query method based on a multi-modal large language model according to claim 4, characterized in that, The overall loss function of the visual fine-tuning stage is: wherein, is an overall loss for the visual fine-tuning phase; is a cross-entropy loss for the classification branch, is a prototype supervised contrastive loss for the contrastive branch, and is a hyperparameter controlling the weight of the cross-entropy loss for the classification branch and the prototype supervised contrastive loss for the contrastive branch.

6. The industrial defect information query method based on a multi-modal large language model according to claim 3, characterized in that, The language fine-tuning in the second stage adopts a self-recurrent manner, and the loss function is a cross-entropy loss based on the multi-turn dialogue dataset, which is used to measure the difference between the predicted distribution of the language model and the real answer label.

7. The industrial defect information query method based on a multi-modal large language model according to claim 6, characterized in that, The cross-entropy loss function based on the multi-turn dialogue dataset is: In the formula, This represents the cross-entropy loss based on the multi-turn dialogue dataset; Indicates the first i The length of the response text in a turn-based dialogue; Indicates the first Before answering the text in a turn-based dialogue One word; Indicates the first Answer the text in the round-robin dialogue One word; express The predicted probability; Indicates alignment with visual embedding; This indicates text embedding.

8. The industrial defect information query method based on a multi-modal large language model according to claim 3, characterized in that, The industrial defect multi-turn dialogue dataset is constructed according to industrial defect analysis knowledge, and the dialogue content revolves around the same defect image to successively query the category, features, root cause and maintenance measures information of the defect.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the industrial defect information query method based on the multi-modal large language model according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the industrial defect information query method based on the multi-modal large language model according to any one of claims 1 to 8.

Images