Industrial quality multi-modal large model prediction method based on multi-source information fusion

Abstract

The invention discloses an industrial quality multi-modal large model prediction method based on multi-source information fusion, and the method comprises the steps: constructing an industrial knowledge graph, generating instruction data containing a process constraint rule based on the industrial knowledge graph, expanding training samples through a multi-modal semantic interpolation and sample generation technology, and generating diversified training samples; establishing a cross-modal feature alignment loss function based on the diversified training samples, and obtaining aligned multi-modal features; based on the aligned multi-modal features, performing three-stage progressive fine tuning on a pre-trained multi-modal large model to obtain a fine-tuned model; and on the basis of the fine-tuned model, constructing a multi-modal decision interpretability framework, and automatically outputting a structured quality diagnosis report by generating an attention thermodynamic diagram, quantizing prediction confidence and combining feature contribution analysis and the industrial knowledge graph. According to the invention, accurate prediction and intelligent diagnosis of industrial process quality are realized.

Description

Technical Field

[0001] This invention relates to the field of industrial process optimization technology, and in particular to a method for predicting industrial quality using a multimodal large model based on multi-source information fusion. Background Technology

[0002] With the deepening of intelligent transformation in my country's manufacturing industry, industrial quality inspection and process optimization have become key links in improving product quality and production efficiency. Resistance spot welding, as a critical connection process widely used in manufacturing, directly affects the structural safety and service performance of products. However, due to the coupling effects of multiple physical fields such as electricity, heat, and force involved in the spot welding process, and the combined influence of material properties, process parameters, and environmental factors, traditional quality monitoring methods are difficult to achieve comprehensive and accurate quality assessment.

[0003] Currently, industrial sites primarily rely on a combination of process parameter monitoring and offline sampling inspection for quality control. This approach has significant limitations: firstly, statistical correlation-based numerical analysis methods struggle to capture the complex nonlinear relationships between process parameters and quality indicators; secondly, destructive sampling inspections are not only costly and time-consuming but also fail to achieve end-to-end quality monitoring. Furthermore, traditional methods often analyze numerical parameters and visual information in isolation, failing to fully leverage the complementarity and synergistic effects between multi-source data.

[0004] In recent years, deep learning technology has brought new technical pathways to industrial quality prediction. By constructing deep neural network models, feature representations can be automatically learned from massive amounts of data, establishing an end-to-end mapping relationship from multi-source inputs to quality indicators. However, this technology still faces significant challenges in practical applications: First, the scarcity of high-quality labeled data restricts model performance, especially in multimodal fusion scenarios, where complete samples of numerical parameters, visual images, and corresponding quality labels need to be obtained simultaneously; second, existing models generally lack the ability to understand industrial knowledge, making it difficult to adapt to complex and ever-changing industrial scenarios; furthermore, the effective fusion and semantic alignment of multimodal data remain technical challenges, and how to coordinate the representation learning process of numerical parameters and visual features has not yet been fully resolved.

[0005] In summary, current technologies lack quality prediction methods that can simultaneously process numerical parameters and visual information, possess industrial knowledge understanding capabilities, and adapt to small sample conditions. Therefore, there is an urgent need to develop an intelligent quality prediction method based on multi-source information fusion to overcome the limitations of traditional statistical methods, achieve accurate prediction and intelligent diagnosis of industrial process quality, and provide reliable technical support for improving the quality and efficiency of the manufacturing industry. Summary of the Invention

[0006] This invention proposes a large-scale multimodal prediction method for industrial quality based on multi-source information fusion, in order to solve key problems in existing quality prediction technologies such as weak multimodal information coordination ability, poor generalization performance under small sample conditions, and lack of industrial knowledge guidance and interpretability support.

[0007] To achieve the above objectives, this invention provides a method for predicting industrial quality using a multimodal large model based on multi-source information fusion, comprising: An industrial knowledge graph containing material properties, process windows, and defect patterns is constructed, and instruction data containing process constraint rules is generated based on the industrial knowledge graph. The training samples are expanded through multimodal semantic interpolation and sample generation techniques to generate diverse training samples. Based on the diverse training samples, a dual-path visual encoder is used to extract macroscopic morphological features and microscopic thermal distribution features respectively. A feature importance self-attention mechanism is introduced to establish a cross-modal feature alignment loss function and obtain aligned multimodal features. Based on the aligned multimodal features, the pre-trained multimodal large model is subjected to three-stage progressive fine-tuning, which successively performs basic multimodal understanding ability activation, multi-task collaborative training, and domain fine adaptation. During the fine-tuning process, a task-aware LoRA rank dynamic adjustment mechanism is adopted to dynamically allocate differentiated rank parameters according to the importance score of the attention head to the quality prediction task, and a hierarchical gradient pruning strategy and weight fusion regularization technique are introduced to obtain the fine-tuned model. Based on the fine-tuned model, a multimodal decision interpretability framework is constructed. By generating an attention heatmap, quantifying prediction confidence, and combining feature contribution analysis with the industrial knowledge graph, a structured quality diagnostic report is automatically output.

[0008] Preferably, generating instruction data containing process constraint rules based on the industrial knowledge graph includes: The entities and relationships in the knowledge graph are embedded and represented using a large model, and the accuracy of the structured knowledge is ensured by using a translation-invariant loss function; process constraint rules are embedded in multimodal dialogue samples in the form of natural language instructions to form an instruction dataset for model training. The process constraint rules are as follows: ; In the formula, The embedding loss of the process constraint rules, , , These represent the head entity, relation, and tail entity in the knowledge graph, respectively. , and These are the embedding vectors for the corresponding entities and relations.

[0009] Preferably, the dual-path visual encoder extracts macroscopic morphological features based on a macroscopic path and extracts microscopic thermal distribution features through a microscopic path. The macroscopic path employs a downsampling strategy to capture the overall geometric structure, while the microscopic path focuses on key regional thermal distribution details through a local attention mechanism. ; In the formula, It is a downsampled image. It is a key region image. This is the output macroscopic morphological feature vector. This is the output microscopic thermal distribution feature vector.

[0010] Preferably, the feature importance self-attention mechanism is as follows: ; In the formula, For attention weight vectors, and These are the learnable weight matrix and bias vector of the self-attention layer. The fused feature vector Attention weights for macroscopic features Attention weights for micro-features; The cross-modal feature alignment loss function is: ; In the formula, For alignment loss, a smaller value indicates that the visual and textual features are closer in semantic space; This is a visual feature projection network used to fuse visual features. Mapped to a cross-modal shared semantic space; This is a text feature projection network used to project process parameter text. Mapped to the same shared semantic space.

[0011] Preferably, the multi-task collaborative training optimizes the balance between regression and classification objectives through dynamic weight adjustment, wherein the dynamic weights are: ; In the formula, For the task The dynamic weights at the current time step. For the task At time step The performance indicators For the task At time step The performance indicators For temperature parameters, The total loss for multi-task collaborative training. For the task The original loss.

[0012] Preferably, the LoRA rank dynamic adjustment mechanism includes: Calculate the Frobenius norm of the gradient activation product of each attention head output, and evaluate the importance score of the Frobenius norm for the quality prediction task; based on the importance score, assign differentiated LoRA ranks to attention heads of different layers through a smoothing mapping function to achieve fine-grained adaptation.

[0013] Preferably, the hierarchical gradient pruning strategy is as follows: ; In the formula, The clipped gradient vector is used for parameter updates; For the cropped first Layer gradient vectors are used for parameter updates; For the first Adaptive pruning threshold for layers; Let L2 be the norm of the gradient vector; The baseline pruning threshold is the base value for all layers. This is the inter-layer scaling factor, which controls the magnitude of the threshold increase with depth; This represents the depth position of the current layer within the network. The total number of layers in the network; The weight fusion regularization technique is as follows: ; In the formula, For the regularization loss of weight fusion, Here are the LoRA weight regularization coefficients. This is the LoRA incremental weight matrix. These are the base weights after fine-tuning. For the original pre-trained weights, This is the weight deviation regularization coefficient.

[0014] Preferably, the attention heatmap is as follows: ; In the formula, To visualize attention heatmaps at location The value at that location reflects the importance of that location for prediction; This is the model's predicted output; For the first Each feature map at location Activation value; This represents the total number of channels. This represents the number of channels in the feature map.

[0015] Preferably, the quantitative prediction confidence level adopts a strategy combining Monte Carlo Dropout and model ensemble, and calculates the variance of the prediction result through several forward propagations, and the variance is negatively correlated with the confidence level.

[0016] Preferably, the feature contribution analysis is as follows: ; In the formula, Features Contribution index, These are the original predicted values. To occlude features The predicted value after that, To assess the total number of samples, For sample index, indicating the first... One sample.

[0017] Compared with the prior art, the present invention has the following advantages and technical effects: (1) This invention constructs a multimodal dataset enhanced with process knowledge, embedding domain expert experience into the model training process in the form of a knowledge graph, thereby enhancing the model’s understanding and reasoning ability of industrial knowledge such as the rationality of process parameters and defect patterns.

[0018] (2) This invention realizes a model fine-tuning mechanism with efficient parameters and task adaptation, achieving high-quality multi-task prediction and rapid domain adaptation under limited labeled data, significantly improving the practicality and stability of the model in real industrial scenarios.

[0019] (3) This invention establishes a quality diagnosis and decision support system with enhanced interpretability, and provides transparent and reliable intelligent decision-making basis for process optimization and quality traceability through multimodal attention visualization, prediction confidence assessment and structured report generation. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a multimodal large model prediction method for industrial quality based on multi-source information fusion, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the model structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the LoRA fine-tuning principle according to an embodiment of the present invention; Figure 4This is a comparison chart of the mean square error of the predicted tensile force and melt core diameter on four models in this embodiment of the invention; Figure 5 This is a schematic diagram of the confusion matrix for quality classification on four models in an embodiment of the present invention; Figure 6 This is a visual and parameter attention heatmap of an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0023] like Figures 1-2 This embodiment proposes a large-scale multi-modal prediction method for industrial quality based on multi-source information fusion, including: An industrial knowledge graph containing material properties, process windows, and defect patterns is constructed, and instruction data containing process constraint rules is generated based on the industrial knowledge graph. The training samples are expanded through multimodal semantic interpolation and sample generation techniques to generate diverse training samples. Based on the diverse training samples, a dual-path visual encoder is used to extract macroscopic morphological features and microscopic thermal distribution features respectively. A feature importance self-attention mechanism is introduced to establish a cross-modal feature alignment loss function and obtain aligned multimodal features. Based on the aligned multimodal features, the pre-trained multimodal large model is subjected to three-stage progressive fine-tuning, which successively performs basic multimodal understanding ability activation, multi-task collaborative training, and domain fine adaptation. During the fine-tuning process, a task-aware LoRA rank dynamic adjustment mechanism is adopted to dynamically allocate differentiated rank parameters according to the importance score of the attention head to the quality prediction task, and a hierarchical gradient pruning strategy and weight fusion regularization technique are introduced to obtain the fine-tuned model. Based on the fine-tuned model, a multimodal decision interpretability framework is constructed. By generating an attention heatmap, quantifying prediction confidence, and combining feature contribution analysis with the industrial knowledge graph, a structured quality diagnostic report is automatically output.

[0024] Furthermore, based on the industrial knowledge graph, instruction data containing process constraint rules is generated, including: The entities and relationships in the knowledge graph are embedded and represented using a large model, and the accuracy of the structured knowledge is ensured by using a translation-invariant loss function. The process constraint rules are embedded into multimodal dialogue samples in the form of natural language instructions to form an instruction dataset for model training.

[0025] Specifically, by constructing a knowledge graph of resistance spot welding process, the experience of domain experts is transformed into a structured knowledge system that machines can understand. The knowledge graph includes three core dimensions: material properties, process windows, and defect patterns, and large-scale models are used for representation learning. Process constraint rules are generated based on the knowledge graph and embedded into instruction dialogues. The process constraint rule embedding loss is: ; In the formula, The embedding loss of the process constraint rules, , , These represent the head entity, relation, and tail entity in the knowledge graph, respectively. , and These are the embedding vectors for the corresponding entities and relations.

[0026] Among these, it is necessary to ensure the head entities in the knowledge graph. ,relation Tail entity Satisfying translation invariance. Establishing a quality verification mechanism for the multimodal instruction dataset. Employing semantic enhancement techniques based on a large multimodal model, the training data is expanded through three methods: multi-granularity instruction reconstruction, cross-semantic interpolation, and multi-view sample generation. The multimodal semantic interpolation enhancement is as follows: ; In the formula, The enhanced text semantic vector represents the continuous transition between the two original texts in the semantic space; The enhanced image features represent the continuous transition between the two original visual samples in the feature space; , This is the original text semantic vector. , Original image features These are the interpolation coefficients. .

[0027] Furthermore, the dual-path visual encoder extracts macroscopic morphological features based on macroscopic paths and microscopic thermal distribution features through microscopic paths. Specifically, the macroscopic path employs a downsampling strategy to capture the overall geometric structure, while the microscopic path focuses on key regional thermal distribution details through a local attention mechanism. ; In the formula, It is a downsampled image. It is a key region image. This is the output macroscopic morphological feature vector. This is the output microscopic thermal distribution feature vector.

[0028] Furthermore, the feature importance self-attention mechanism is as follows: ; In the formula, For attention weight vectors, and These are the learnable weight matrix and bias vector of the self-attention layer. The fused feature vector Attention weights for macroscopic features Attention weights for micro-features; The cross-modal feature alignment loss function is: ; In the formula, For alignment loss, a smaller value indicates that the visual and textual features are closer in semantic space; This is a visual feature projection network used to fuse visual features. Mapped to a cross-modal shared semantic space; This is a text feature projection network used to project process parameter text. Mapped to the same shared semantic space.

[0029] and Visual features and text parameters are projected into the same semantic space, respectively. Calculate cosine similarity to ensure consistency between visual features and numerical parameters in the semantic space.

[0030] Furthermore, a three-stage progressive fine-tuning strategy is adopted to gradually improve the model's performance on industrial quality prediction tasks. The first stage awakens the model's basic multimodal understanding capabilities through a general visual language task, establishing an initial mapping of multimodal inputs.

[0031] The second phase involves multi-task collaborative training, optimizing the regression and classification objectives through dynamic weight adjustments: ; In the formula, For the task The dynamic weights at the current time step. For the task At time step The performance indicators For the task At time step The performance indicators The total loss for multi-task collaborative training. For the task The original loss, Temperature is used as a parameter to control the smoothness of the weight distribution. Dynamic weights allow the model to focus on tasks with greater potential for improvement and balance the learning progress of different prediction targets.

[0032] The third stage employs a course-based learning strategy, progressively training from simple to complex samples to achieve fine-grained domain adaptation. The difficulty level of the course learning is scheduled as follows: ; In the formula, For time step The sample difficulty threshold, For scheduling coefficients, The minimum difficulty threshold, The maximum difficulty threshold, For the total number of training steps, exponential scheduling ensures a smooth transition in difficulty, avoiding training instability and focusing on the specific needs of industrial quality prediction. Finally, progressive knowledge distillation maintains model stability. ; In the formula, The knowledge distillation loss term measures the difference in the distribution of the student model's and the teacher model's outputs. This refers to the distillation temperature. KL divergence measures the difference between two probability distributions. and The differences between them; The Softmax function converts logits into a probability distribution. The outputs are the logits for the student model and the teacher model, respectively. The weighting factor for time decay is... Over time, the reliance on teacher models will gradually decrease. The initial maximum weight, This represents the total number of training steps.

[0033] Furthermore, the LoRA rank dynamic adjustment mechanism includes: Calculate the Frobenius norm of the gradient activation product of each attention head output, and evaluate the importance score of the Frobenius norm for the quality prediction task; based on the importance score, assign differentiated LoRA ranks to attention heads of different layers through a smoothing mapping function to achieve fine-grained adaptation.

[0034] Specifically, a task-aware LoRA rank dynamic adjustment mechanism is designed, and the LoRA principle is as follows: Figure 3 As shown, differentiated rank parameters are assigned based on the importance scores of each attention head, and the contribution of each attention head to the prediction task is calculated: ; In the formula, For the first The importance score of the layer For the total number of attention heads, For the task loss function, Let be the gradient matrix of the loss function for the output of the attention head. For the first Layer The output of each attention head; Importance is evaluated using the Frobenius norm based on the gradient activation product, and the LoRA rank assignment is guided by the importance score: ; In the formula, For the first The LoRA rank of the layer, in Within the range; Use the sigmoid function to achieve smooth mapping; By controlling the sensitivity of allocation, important heads obtain higher rank, achieving optimal resource allocation; It is the minimum value of the rank. The maximum value of rank. The minimum importance score across all layers. This represents the maximum importance score across all layers.

[0035] Furthermore, the hierarchical gradient pruning strategy is as follows: ; In the formula, The clipped gradient vector is used for parameter updates; For the cropped first Layer gradient vectors are used for parameter updates; For the first Adaptive pruning threshold for layers; Let L2 be the norm of the gradient vector; The baseline pruning threshold is the base value for all layers. This is the inter-layer scaling factor, which controls the magnitude of the threshold increase with depth; This represents the depth position of the current layer within the network. The total number of layers in the network; Higher-level networks typically have larger gradients, so the pruning threshold increases with the number of layers. Different pruning thresholds are used for the gradients of different layers to prevent gradient explosion during fine-tuning.

[0036] The weight fusion regularization technique is as follows: ; In the formula, For the regularization loss of weight fusion, Here are the LoRA weight regularization coefficients. This is the LoRA incremental weight matrix. These are the base weights after fine-tuning. For the original pre-trained weights, This is the weight deviation regularization coefficient.

[0037] L2 regularization ensures smooth integration of the adapter and backbone models, balancing the compactness of adapter parameters with the stability of backbone parameters.

[0038] Furthermore, the attention heatmap is as follows: Figure 6 As shown, the expression is: ; In the formula, To visualize attention heatmaps at location The value at a given location reflects the importance of that location for prediction. This is the model's predicted output. For the first Each feature map at location activation value, The total number of channels. This represents the number of channels in the feature map.

[0039] Furthermore, the quantitative prediction confidence level adopts a strategy combining Monte Carlo Dropout and model ensemble, and calculates the variance of the prediction result through several forward propagations, and the variance is negatively correlated with the confidence level.

[0040] Specifically, the reliability of quantitative prediction based on Monte Carlo Dropout and ensemble strategies can be expressed as: ; In the formula, This represents the total forecast variance. For sampling index, For Monte Carlo Dropout sampling times, For the first Model prediction output after Dropout sampling This is the final predicted value. For the first The variance estimated by the model itself at each sampling time is based on R Sub-Monte Carlo Dropout sampling calculates the predicted distribution. The first term represents the inter-model difference, and the second term represents the intra-model difference. The confidence level is negatively correlated with the variance.

[0041] Furthermore, a structured diagnostic report automatic generation system is designed, which combines feature contribution analysis and process knowledge graph to output industrial-grade quality analysis results. The feature contribution analysis is as follows: ; In the formula, Features Contribution index, These are the original predicted values. To occlude features The predicted value after that, To assess the total number of samples, For sample index, indicating the first... One sample.

[0042] To more clearly illustrate the technical solution of the present invention, specific embodiments are provided below for description: 1. Experimental data: The resistance spot welding process dataset used in this experiment contains multiple complete samples, and the system records multi-source information during the welding process. Process parameters cover key settings such as electrode pressure, welding current, welding time, and electrode angle; material properties include combinations of metal plates of different thicknesses and types. Quality evaluation strictly follows the AWS D8-9 standard, providing quantitative indicators such as tensile shear strength and weld nugget diameter. The core innovation of the dataset lies in the simultaneous acquisition of surface morphology images and infrared thermograms of each weld point, constructing a complete visual and thermodynamic multimodal data system. This dataset covers various typical process states such as normal welding, spatter, and lack of fusion, providing a high-quality data foundation for supervised learning-based welding quality prediction, defect identification, and multimodal feature correlation analysis. Table 1 describes the input and output parameters and the number of samples.

[0043] Table 1 2. Method Validation: To systematically evaluate the effectiveness of the method in this embodiment, a rigorous multimodal large-scale model comparison experimental framework was constructed. The experimental design strictly followed the principle of controlling variables to ensure that each comparison model was trained and its performance evaluated under completely consistent experimental conditions.

[0044] In terms of dataset configuration, a multimodal benchmark dataset for resistance spot welding process is adopted. This dataset systematically integrates multi-source information such as process parameters, surface morphology images, and infrared thermograms. The training set and test set are divided in an 8:2 ratio, and a stratified sampling strategy is used to ensure the consistency of the distribution of samples of each category in the training set and test set, effectively eliminating the interference of data bias on performance evaluation.

[0045] In terms of comparison model selection, four representative open-source multimodal large models were chosen as benchmark comparison objects, including the lightweight representative Qwen2.5-VL-3B-Instruct, the upgraded version Qwen3-VL-8B-Instruct, the efficient quantization model Deepseek-vl-7b-chat-4bit, and InternVL3_5-8B, which specializes in visual language tasks.

[0046] The training strategy employs a unified configuration, with all models based on a supervised fine-tuning paradigm and trained using the same multimodal instruction dataset. During fine-tuning, the LoRA parameter is uniformly set to rank r=8 and scaling factor α=16; the optimizer configuration uses AdamW with an initial learning rate of 1e-4, coupled with a cosine annealing scheduling strategy; the training epochs are uniformly set to 10, the batch size to 1, and the gradient accumulation step count to 8, while an early stopping strategy is employed to prevent overfitting. The evaluation metrics cover both regression and classification tasks. For regression tasks, mean squared error (MSE), mean absolute error (MAE), coefficient of determination (R²), and mean relative error are used for quantitative evaluation. For classification tasks, accuracy, F1 score, and recall are used for comprehensive evaluation to ensure the comprehensiveness and reliability of performance assessment.

[0047] Table 2 Table 3 In the tensile test prediction task, all four multimodal large models demonstrated excellent predictive performance, as shown in Table 2. Among them, the InternVL3_5-8B model performed best, with a mean squared error of 17439.67, a mean absolute error of 124.70N, a coefficient of determination of 0.98, and a mean relative error of only 3.76%, making it the best among all compared models. The performance of Qwen3-VL-8B-Instruct and Deepseek-vl-7b-chat-4bit was quite similar, with a coefficient of determination of 0.98 and mean absolute errors of 127.57N and 127.30N, respectively. Relatively speaking, the performance of the Qwen2.5-VL-3B-Instruct model was slightly inferior, but its coefficient of determination still reached 0.97, demonstrating good predictive ability.

[0048] In the task of predicting the melting nucleus diameter, the performance of each model varied significantly, as shown in Table 3. The Qwen3-VL-8B-Instruct model performed best, with a mean squared error of 0.13, a mean absolute error of 0.35 mm, a coefficient of determination of 0.53, and a mean relative error of 9.62%. The Deepseek-vl-7b-chat-4bit and Qwen2.5-VL-3B-Instruct models performed similarly, with coefficients of determination of 0.42 and 0.38, respectively. Notably, the InternVL3_5-8B model performed relatively poorly in this task, with a coefficient of determination of only 0.25 and a mean relative error of 12.30%, indicating certain limitations in the geometric dimension prediction task. Figure 4 The mean square error of the four models in predicting tensile force and melt diameter is shown respectively.

[0049] Table 4 According to the experimental results in Table 4, all four multimodal large models demonstrated excellent performance in the welding quality classification task. The Qwen3-VL-8B-Instruct model performed the best, achieving an accuracy of 98.69%, a recall of 99.06%, and an F1 score of 90.87%, ranking first in all evaluation metrics. The InternVL3_5-8B model followed closely behind, with an accuracy of 97.73%, a recall of 98.25%, and an F1 score of 85.67%, also showing excellent performance. The Deepseek-vl-7b-chat-4bit model performed well in terms of accuracy (96.42%) and recall (97.30%), but its F1 score (79.95%) was relatively low. Although the Qwen2.5-VL-3B-Instruct model performed reasonably well in terms of accuracy (95.11%) and recall (96.36%), its F1 score (75.29%) was significantly lower than other models, indicating that there is room for improvement in terms of positive and negative sample balance. Figure 5 The confusion matrix of different models in quality classification is shown.

[0050] In summary, this experiment fully validates the effectiveness of multimodal large models in industrial quality prediction tasks. By deeply fusing numerical parameters and visual information, this method lays a robust foundation for building intelligent industrial quality diagnostic systems. Furthermore, the performance differences exhibited by the model across different types of prediction tasks provide a clear direction for subsequent targeted model optimization and research.

[0051] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An industrial quality multi-modal large model prediction method based on multi-source information fusion, characterized in that, include: An industrial knowledge graph containing material properties, process windows, and defect patterns is constructed, and instruction data containing process constraint rules is generated based on the industrial knowledge graph. The training samples are expanded through multimodal semantic interpolation and sample generation techniques to generate diverse training samples. Based on the diverse training samples, a dual-path visual encoder is used to extract macroscopic morphological features and microscopic thermal distribution features respectively. A feature importance self-attention mechanism is introduced to establish a cross-modal feature alignment loss function and obtain aligned multimodal features. Based on the aligned multimodal features, the pre-trained multimodal large model is subjected to three-stage progressive fine-tuning, which successively performs basic multimodal understanding ability activation, multi-task collaborative training, and domain fine adaptation. During the fine-tuning process, a task-aware LoRA rank dynamic adjustment mechanism is adopted to dynamically allocate differentiated rank parameters according to the importance score of the attention head to the quality prediction task, and a hierarchical gradient pruning strategy and weight fusion regularization technique are introduced to obtain the fine-tuned model. Based on the fine-tuned model, a multimodal decision interpretability framework is constructed. By generating an attention heatmap, quantifying prediction confidence, and combining feature contribution analysis with the industrial knowledge graph, a structured quality diagnostic report is automatically output.

2. The industrial quality multi-modal large model prediction method based on multi-source information fusion according to claim 1, characterized in that, Based on the industrial knowledge graph, instruction data containing process constraint rules is generated, including: The entities and relationships in the knowledge graph are embedded and represented using a large model, and the accuracy of the structured knowledge is ensured by using a translation-invariant loss function; process constraint rules are embedded in multimodal dialogue samples in the form of natural language instructions to form an instruction dataset for model training. The process constraint rules are as follows: ； wherein, the embedding loss for the process constraint rules, , , head entity, relation and tail entity in the knowledge graph, respectively, , and the embedding vectors for the corresponding entities and relations.

3. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 1, characterized in that, The dual-path visual encoder extracts macroscopic morphological features based on a macroscopic path and microscopic thermal distribution features through a microscopic path. The macroscopic path employs a downsampling strategy to capture the overall geometric structure, while the microscopic path focuses on key regional thermal distribution details through a local attention mechanism. ； In the formula, It is a downsampled image. It is a key region image. This is the output macroscopic morphological feature vector. This is the output microscopic thermal distribution feature vector.

4. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 3, characterized in that, The feature importance self-attention mechanism is as follows: ； In the formula, For attention weight vectors, and These are the learnable weight matrix and bias vector of the self-attention layer. The fused feature vector Attention weights for macroscopic features Attention weights for micro-features; The cross-modal feature alignment loss function is: ； In the formula, For alignment loss, a smaller value indicates that the visual and textual features are closer in semantic space; This is a visual feature projection network used to fuse visual features. Mapped to a cross-modal shared semantic space; This is a text feature projection network used to project process parameter text. Mapped to the same shared semantic space.

5. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 1, characterized in that, The multi-task collaborative training optimizes the balance between regression and classification objectives through dynamic weight adjustment, wherein the dynamic weights are: ； In the formula, For the task The dynamic weights at the current time step. For the task At time step The performance indicators For the task At time step The performance indicators For temperature parameters, The total loss for multi-task collaborative training. For the task The original loss.

6. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 1, characterized in that, The LoRA rank dynamic adjustment mechanism includes: Calculate the Frobenius norm of the gradient activation product of each attention head output, and evaluate the importance score of the Frobenius norm for the quality prediction task; based on the importance score, assign differentiated LoRA ranks to attention heads of different layers through a smoothing mapping function to achieve fine-grained adaptation.

7. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 6, characterized in that, The hierarchical gradient pruning strategy is as follows: ； In the formula, The clipped gradient vector is used for parameter updates; For the cropped first Layer gradient vectors are used for parameter updates; For the first Adaptive pruning threshold for layers; Let L2 be the norm of the gradient vector; The baseline pruning threshold is the base value for all layers. This is the inter-layer scaling factor, which controls the magnitude of the threshold increase with depth; This represents the depth position of the current layer within the network. The total number of layers in the network; The weight fusion regularization technique is as follows: ； In the formula, For the regularization loss of weight fusion, Here are the LoRA weight regularization coefficients. This is the LoRA incremental weight matrix. These are the base weights after fine-tuning. For the original pre-trained weights, This is the weight deviation regularization coefficient.

8. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 1, characterized in that, The attention heatmap is as follows: ； In the formula, To visualize attention heatmaps at location The value at that location reflects the importance of that location for prediction; This is the model's predicted output; For the first Each feature map at location Activation value; This represents the total number of channels. This represents the number of channels in the feature map.

9. The industrial quality multimodal large model prediction method based on multi-source information fusion according to claim 8, characterized in that, The quantitative prediction confidence level adopts a strategy combining Monte Carlo Dropout and model ensemble, and calculates the variance of the prediction result through several forward propagations, and the variance is negatively correlated with the confidence level.

10. The method for predicting industrial quality using a multi-modal large model based on multi-source information fusion according to claim 1, characterized in that, The feature contribution analysis is as follows: ； In the formula, Features Contribution index, These are the original predicted values. To occlude features The predicted value after that, To assess the total number of samples, For sample index, indicating the first... One sample.

Images