Industrial equipment fault diagnosis large language model multi-expert LoRA fine tuning method

Abstract

The invention discloses an industrial equipment fault diagnosis large language model multi-expert LoRA fine tuning method, which comprises the following steps: S1, data collection and preprocessing: collecting data related to industrial equipment fault diagnosis, cleaning and labeling a data set, and dividing the data set into a training set, a verification set and a test set; s2, selecting a basic large language model; s3, constructing a multi-expert LoRA module, respectively constructing expert LoRA modules for different industrial equipment fields, and designing an independent dynamic routing fine tuning strategy; s4, performing multi-expert LoRA fine tuning, inputting the training set into the basic large language model, activating all expert LoRA modules, and updating part of parameters of the basic large language model; and S5, model fusion and evaluation: fusing the parameters of all expert LoRA modules with the parameters of the basic large language model, and the industrial equipment fault diagnosis large language model multi-expert LoRA fine tuning method overcomes the problems of large calculation amount, over-fitting and difficulty in fully utilizing multi-field expert knowledge in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and industrial equipment fault diagnosis. Specifically, it relates to a multi-expert LoRA fine-tuning method for large language models in industrial equipment fault diagnosis. Background Art

[0002] In industrial production, the stable operation of industrial equipment is crucial. Equipment failures not only cause production interruptions but may also trigger safety accidents, resulting in huge economic losses. Traditional industrial equipment fault diagnosis methods mainly rely on manual experience and professional knowledge, suffering from problems such as low diagnosis efficiency and poor accuracy. With the development of artificial intelligence technology, large language models have achieved remarkable results in fields such as natural language processing. Applying large language models to industrial equipment fault diagnosis can improve the efficiency and accuracy of diagnosis. However, directly using general large language models for industrial equipment fault diagnosis has certain limitations because general large language models lack professional knowledge in the field of industrial equipment. To make large language models better adapt to industrial equipment fault diagnosis tasks, fine-tuning is required.

[0003] Existing fine-tuning methods usually update parameters across the entire model. This method has a large computational amount, takes a long time, and is prone to overfitting problems. In addition, industrial equipment fault diagnosis involves professional knowledge in multiple fields, and a single fine-tuning method is difficult to fully utilize the knowledge of experts in different fields.

[0004] Therefore, providing a multi-expert LoRA fine-tuning method for large language models in industrial equipment fault diagnosis that only updates some parameters of the model during use, greatly reducing the computational amount and time cost, and can fully utilize the expert knowledge in multiple fields by independently fine-tuning expert LoRA modules in different fields, and avoiding the overfitting problems easily caused by a single fine-tuning method is an urgent problem to be solved by the present invention. Summary of the Invention

[0005] To address the above technical problems, the objective of the present invention is to overcome the problems of large computational amount, overfitting, and difficulty in fully utilizing the knowledge of experts in multiple fields existing in the prior art. Thus, a multi-expert LoRA fine-tuning method for large language models in industrial equipment fault diagnosis is provided. During use, only some parameters of the model are updated, greatly reducing the computational amount and time cost. By independently fine-tuning expert LoRA modules in different fields, the expert knowledge in multiple fields can be fully utilized, and the overfitting problems easily caused by a single fine-tuning method can be avoided.

[0006] To achieve the above objective, the present invention provides a multi-expert LoRA fine-tuning method for large language models in industrial equipment fault diagnosis, including the following steps:

[0007] S1. Data collection and preprocessing: Collect data related to industrial equipment fault diagnosis, construct a heterogeneous dataset, clean and annotate the dataset, and divide it into a training set, a validation set, and a test set;

[0008] S2. Select a basic large language model;

[0009] S3. Construct a multi-expert LoRA module, construct expert LoRA modules for different industrial equipment fields respectively, and design an independent dynamic routing fine-tuning strategy;

[0010] S4. Multi-expert LoRA fine-tuning: Input the training set into the basic large language model, activate all expert LoRA modules, and update some parameters of the basic large language model;

[0011] S5. Model fusion and evaluation: Integrate the parameters of all expert LoRA modules with the parameters of the basic large language model, and evaluate the fused model using the validation set and the test set.

[0012] Preferably, in the step S1, the dataset includes: equipment operation parameters, fault descriptions, maintenance records, time-series sensor data, and image data; where,

[0013] The time-series sensor data includes: vibration and temperature data;

[0014] The image data includes: infrared imaging and mechanical component image data.

[0015] Preferably, in the step S1, the specific method for cleaning and annotating the dataset includes the following steps:

[0016] S101. Data collection: Collect a large amount of industrial equipment fault diagnosis data from multiple industrial enterprises, including operation parameters, fault descriptions, and maintenance records of mechanical equipment, electrical equipment, and hydraulic equipment, etc.;

[0017] S102. Data cleaning: Clean the collected data to remove noise data and duplicate data;

[0018] S103. Data annotation: Annotate each piece of data with a fault type and severity level, which are divided into three categories: minor fault, moderate fault, and severe fault.

[0019] S104. Data division: Divide the annotated data into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0020] Preferably, in the step S2, the basic large language model is GPT, BERTCodeLlama-7B, or Qwen-1.8B.

[0021] Preferably, in the step S3, each expert LoRA module includes a low-rank matrix pair; where

[0022] the low-rank matrix pair includes: A ∈ R d×r and B ∈ R r×d , r = 8, which is inserted into the attention layer and the FFN layer of the base large language model.

[0023] Preferably, in the step S3, the specific method for designing an independent dynamic routing fine-tuning strategy includes the following steps:

[0024] S301. Design a gating network, input multi-modal fusion features, and output an expert weight vector α i , and generate the final adaptation parameters through weighted summation:

[0025]

[0026] S302. The routing network adopts a lightweight MLP structure, and the Gumbel-Softmax is used to optimize the discrete selection process during training.

[0027] Preferably, in the step S5, the method for fusing the parameters of all expert LoRA modules with the parameters of the base large language model is:

[0028] Superimpose the incremental parameters ΔW of each expert LoRA on the base large language model to generate the final inference model:

[0029] W final = W base + ∑ΔW i

[0030] Preferably, in the step S4, the loss function is calculated by the backpropagation algorithm and the parameters of the expert LoRA module are updated.

[0031] Preferably, in the step S5, the evaluation metrics include: accuracy, recall rate, and F1 value.

[0032] Preferably, the calculation method of the F1 value is:

[0033] The calculation of the F1 value is based on precision and recall. Precision refers to the proportion of true positive samples among the samples predicted as positive by the model, and recall refers to the proportion of samples that are truly positive and are correctly predicted as positive; therefore, the calculation formula of the F1 value is:

[0034]

[0035] where

[0036] According to the above technical solution, the beneficial effects of the multi-expert LoRA fine-tuning method for the industrial equipment fault diagnosis large language model provided by the present invention when in use are as follows:

[0037] (1) The present invention adopts the LoRA fine-tuning method, which only updates some parameters of the model, greatly reducing the computational amount and time cost.

[0038] (2) Avoid overfitting: By independently fine-tuning the expert LoRA modules in different fields, the present invention can make full use of the expert knowledge in multiple fields and avoid the overfitting problem easily caused by a single fine-tuning method.

[0039] (3) Improve diagnostic accuracy: The multi-expert LoRA module of the present invention can better adapt to the characteristics and requirements of different industrial equipment fields, improving the accuracy and reliability of industrial equipment fault diagnosis.

[0040] Other features and advantages of the present invention will be described in detail in the subsequent specific implementation part; moreover, the parts not involved in the present invention are the same as or can adopt the existing technology. Brief Description of the Drawings

[0041] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification, and are used to explain the present invention together with the following specific implementation manners, but do not constitute a limitation to the present invention. In the drawings:

[0042] Figure 1 is a flowchart of the multi-expert LoRA fine-tuning method for the industrial equipment fault diagnosis large language model provided in a preferred implementation manner of the present invention. Specific Implementation Manner

[0043] The following details the specific implementation manners of the present invention with reference to the drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0044] As Figure 1 shown, the multi-expert LoRA fine-tuning method for the industrial equipment fault diagnosis large language model provided by the present invention includes the following steps:

[0045] S1. Data collection and preprocessing: Collect data related to industrial equipment fault diagnosis, construct a heterogeneous data set, clean and label the data set, and divide it into a training set, a validation set, and a test set;

[0046] S2. Select a basic large language model;

[0047] S3. Construct a multi-expert LoRA module, respectively construct expert LoRA modules for different industrial equipment fields, and design an independent dynamic routing fine-tuning strategy;

[0048] S4. Multi-expert LoRA fine-tuning: Input the training set into the base large language model, activate all expert LoRA modules, and update some parameters of the base large language model.

[0049] S5. Model fusion and evaluation: Integrate the parameters of all expert LoRA modules with the parameters of the base large language model, and evaluate the fused model using the validation set and the test set.

[0050] In the above solution, in step S1, standardize the time series signal, perform sliding window segmentation and frequency domain feature extraction (FFT, wavelet transform); construct a fault knowledge graph for text data through entity recognition and align it with the timestamps of sensor data.

[0051] In step S3: Constructing multi-expert LoRA modules, for different industrial equipment fields (such as machinery, electricity, hydraulics, etc.), construct expert LoRA modules respectively. Each expert LoRA module contains a low-rank matrix pair for fine-tuning some parameters of the base large language model. Design an independent fine-tuning strategy for each expert LoRA module, and adjust the fine-tuning parameters and learning rate according to the characteristics and requirements of this field. Therefore, the present invention can better adapt to the characteristics and requirements of different industrial equipment fields through multi-expert LoRA modules, improving the accuracy and reliability of industrial equipment fault diagnosis.

[0052] In step S4, the multi-expert LoRA fine-tuning process adopts staged training:

[0053] Stage 1: Freeze the base model and only train the expert LoRA adapter, and use Focal Loss to solve the class imbalance problem.

[0054] Stage 2: Jointly optimize the expert adapter and the routing network, and introduce a reinforcement learning reward mechanism (such as diagnostic accuracy) to dynamically adjust the expert weights.

[0055] Efficient parameter update: Only update the LoRA adapter parameters (less than 3% of the total parameters), and the video memory requirement for single-device training is reduced to <16GB.

[0056] In the above solution, the LoRA fine-tuning method is adopted to only update some parameters of the model. Compared with full-parameter fine-tuning, the training time is reduced by 50%-70%, greatly reducing the computational amount and time cost. In addition, by independently fine-tuning the expert LoRA modules in different fields, the expert knowledge in multiple fields can be fully utilized, avoiding the overfitting problem easily caused by a single fine-tuning method.

[0057] In a preferred embodiment of the present invention, in the step S1, the data set includes: device operation parameters, fault descriptions, maintenance records, time series sensor data, and image data; where

[0058] the time series sensor data includes: vibration and temperature data;

[0059] the image data includes: infrared imaging and mechanical component image data.

[0060] In a preferred embodiment of the present invention, in the step S1, the specific method for cleaning and annotating the data set includes the following steps:

[0061] S101. Data collection: A large amount of industrial equipment fault diagnosis data is collected from multiple industrial enterprises, including operation parameters, fault descriptions, and maintenance records of mechanical equipment, electrical equipment, and hydraulic equipment, etc.;

[0062] S102. Data cleaning: The collected data is cleaned to remove noise data and duplicate data;

[0063] S103. Data annotation: Each piece of data is annotated with a fault type and severity level, which are divided into three categories: minor faults, moderate faults, and severe faults.

[0064] S104. Data division: The annotated data is divided into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0065] In a preferred embodiment of the present invention, in the step S2, the basic large language model is GPT, BERT, CodeLlama-7B, or Qwen-1.8B.

[0066] In the above solution, GPT-3 is selected as the basic large language model, and this model has high performance in natural language processing tasks.

[0067] In a preferred embodiment of the present invention, in the step S3, each expert LoRA module contains a low-rank matrix pair; where

[0068] the low-rank matrix pair includes: A ∈ R d×r and B ∈ R r×d , r = 8, which is inserted into the attention layer and FFN layer of the basic large language model.

[0069] In the above solution, a low-rank matrix is used to fine-tune some parameters of the basic large language model. An independent fine-tuning strategy is designed for each expert LoRA module, and according to the characteristics and requirements of this field, the fine-tuning parameters and learning rate are adjusted. Among them, a parameter isolation mechanism is adopted among experts to avoid interference between tasks. For example, for different industrial equipment fields (such as machinery, electricity, hydraulics, etc.), expert LoRA modules are constructed separately. Each expert LoRA module contains a pair of low-rank matrices for fine-tuning some parameters of the basic large language model.

[0070] In a preferred embodiment of the present invention, in step S3, the specific method for designing an independent dynamic routing fine-tuning strategy includes the following steps:

[0071] S301. Design a gating network, input multi-modal fusion features, and output an expert weight vector α i , and generate the final adaptation parameter through weighted summation:

[0072]

[0073] S302. The routing network adopts a lightweight MLP structure, and the Gumbel-Softmax is used to optimize the discrete selection process during training.

[0074] In a preferred embodiment of the present invention, in step S5, the method for fusing the parameters of all expert LoRA modules with the parameters of the basic large language model is:

[0075] Superimpose the incremental parameter ΔW of each expert LoRA on the basic large language model to generate a final inference model:

[0076] W final = W base + ∑ΔW i

[0077] In a preferred embodiment of the present invention, in step S4, the loss function is calculated through the backpropagation algorithm and the parameters of the expert LoRA module are updated.

[0078] [[ID=3**4]]In the above solution, through the backpropagation algorithm, the loss function is calculated and the parameters of the expert LoRA module are updated, so that the model can better adapt to the industrial equipment fault diagnosis task.

[0079] In a preferred embodiment of the present invention, in step S5, the evaluation indexes include: accuracy rate, recall rate, and F1 value.

[0080] In the above solution, the F1 value is a metric used to evaluate the performance of a model. It combines precision and recall and is commonly used in classification tasks, especially important when dealing with imbalanced datasets. For example, the F1 value is very useful in scenarios where precision and recall need to be balanced, such as in medical diagnosis, fraud detection, and natural language processing. In medical diagnosis, the F1 value can help evaluate the accuracy of diagnostic tests, ensuring both the correct identification of diseased individuals (high recall) and the avoidance of misclassifying healthy individuals as diseased (high precision).

[0081] In addition, the evaluation metrics are divided into:

[0082] Task-level metrics: Fault classification F1 score, root cause localization accuracy;

[0083] Efficiency metrics: Single-sample inference latency (<200ms), expert activation sparsity (only 1-2 expert weights > 0.5).

[0084] In a preferred embodiment of the present invention, the calculation method of the F1 value is:

[0085] The calculation of the F1 value is based on precision and recall. Precision refers to the proportion of true positive predictions among the positive predictions made by the model, and recall refers to the proportion of truly positive samples that are correctly predicted as positive. Therefore, the formula for calculating the F1 value is:

[0086]

[0087] Where,

[0088] In the above solution, Precision is precision, Recall is recall, and the F1 value ranges from 0 to 1. The closer the value is to 1, the better the model performance. It provides a comprehensive evaluation perspective and avoids the potential misguidance of a single metric (such as accuracy), especially when dealing with class imbalance problems.

[0089] In summary, the advantages of the present invention include the following:

[0090] (1) Computational efficiency: Compared with full-parameter fine-tuning, the training time is reduced by 50%-70%;

[0091] (2) Multi-task compatibility: Supports the joint diagnosis of multiple types of faults such as mechanical, electrical, and software, with an 8%-12% increase in the F1 score;

[0092] (3) Dynamic expansion ability: When adding an expert module, only the routing network needs to be fine-tuned, and the parameter increment < 0.5%.

[0093] In summary, the multi-expert LoRA fine-tuning method for the industrial equipment fault diagnosis large language model provided by the present invention overcomes the problems of large computational amount, overfitting, and difficulty in fully utilizing the knowledge of multi-domain experts existing in the prior art.

[0094] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0095] In addition, it should be noted that, in the case of no contradiction, the various specific technical features described in the above specific embodiments can be combined in any suitable manner. To avoid unnecessary repetition, the present invention will not separately describe various possible combination methods.

[0096] Furthermore, any combination can be made between various different embodiments of the present invention as long as it does not violate the idea of the present invention, and it should also be regarded as the content disclosed by the present invention.

Claims

1. A multi-expert LoRA fine-tuning method for large language models in industrial equipment fault diagnosis, characterized in that, It includes the following steps: S1. Data collection and preprocessing: Collect data related to industrial equipment fault diagnosis, construct a heterogeneous dataset, clean and annotate the dataset, and divide it into a training set, a validation set, and a test set; S2. Select a basic large language model; S3. Construct a multi-expert LoRA module, construct expert LoRA modules for different industrial equipment fields respectively, and design an independent dynamic routing fine-tuning strategy; S4. Multi-expert LoRA fine-tuning: Input the training set into the basic large language model, activate all expert LoRA modules, and update some parameters of the basic large language model; S5. Model fusion and evaluation: Integrate the parameters of all expert LoRA modules with the parameters of the basic large language model, and evaluate the fused model using the validation set and the test set.

2. The multi-expert LoRA fine-tuning method for the large language model of industrial equipment fault diagnosis according to claim 1, wherein, In step S1, the dataset includes: equipment operation parameters, fault descriptions, maintenance records, time-series sensor data, and image data; among them, the time-series sensor data includes: vibration and temperature data; the image data includes: infrared imaging and mechanical component image data.

3. The multi-expert LoRA fine-tuning method for the large language model of industrial equipment fault diagnosis according to claim 1, wherein, In step S1, the specific method for cleaning and annotating the dataset includes the following steps: S101. Data collection: Collect a large amount of industrial equipment fault diagnosis data from multiple industrial enterprises, including operation parameters, fault descriptions, and maintenance records of mechanical equipment, electrical equipment, and hydraulic equipment, etc.; S102. Data cleaning: Clean the collected data to remove noise data and duplicate data; S103. Data annotation: Annotate each piece of data with a fault type and severity level, which are divided into three categories: minor faults, moderate faults, and severe faults. S104. Data division: Divide the annotated data into a training set, a validation set, and a test set according to a ratio of 7:2:

1.

4. The multi-expert LoRA fine-tuning method for the large language model of industrial equipment fault diagnosis according to claim 1, characterized in that, In step S2, the basic large language model is GPT, BERTCodeLlama-7B, or Qwen-1.8B.

5. The method for multi-expert LoRA fine-tuning of the large language model for industrial equipment fault diagnosis according to claim 1, wherein In step S3, each expert LoRA module contains a low-rank matrix pair; among them, The low-rank matrix pair includes: A ∈ R d×r and B ∈ R r×d , r = 8, inserted into the attention layer and the FFN layer of the base large language model.

6. The multi-expert LoRA fine-tuning method for the large language model of industrial equipment fault diagnosis according to claim 1 or 5, characterized in that In step S3, the specific method for designing an independent dynamic routing fine-tuning strategy includes the following steps: S301. Design a gating network, input the multi-modal fusion features, and output the expert weight vector α i , and generate the final adaptation parameter through weighted summation: S302. The routing network adopts a lightweight MLP structure, and the Gumbel-Softmax is used to optimize the discrete selection process during training.

7. The multi-expert LoRA fine-tuning method for the large language model of industrial equipment fault diagnosis according to claim 1, characterized in that, In step S5, the method for integrating the parameters of all expert LoRA modules with the parameters of the basic large language model is: Superimpose the incremental parameter ΔW of each expert LoRA onto the basic large language model to generate the final inference model: W final = W base + ∑ΔW i。 8. The method for multi-expert LoRA fine-tuning of the large language model for industrial equipment fault diagnosis according to claim 1, characterized in that In step S4, the loss function is calculated through the backpropagation algorithm and the parameters of the expert LoRA module are updated.

9. The method for multi-expert LoRA fine-tuning of the large language model for industrial equipment fault diagnosis according to claim 1, wherein In step S5, the evaluation metrics include: accuracy, recall rate, and F1 value.

10. The method for multi-expert LoRA fine-tuning of the large language model for industrial equipment fault diagnosis according to claim 9, characterized in that The calculation method of the F1 value is: The calculation of the F1 value is based on the precision rate and the recall rate. The precision rate refers to the proportion of truly positive samples among the samples predicted as positive by the model, and the recall rate refers to the proportion of samples that are truly positive and are correctly predicted as positive; therefore, the calculation formula of the F1 value is: Among them, 。

Images