A hybrid expert efficient fine-tuning routing method for coal mine equipment operation and maintenance multi-modal large model

Abstract

A kind of mixed expert efficient fine-tuning routing method for coal mine equipment operation and maintenance multi-modal large model, steps include: coal mine equipment multi-source heterogeneous data acquisition;Multi-modal data feature extraction and joint embedding;Hierarchical routing and expert probability calculation;Route stability smoothing processing;Self-adaptive Top-k sparse expert activation and load constraint;Mixed expert efficient fine-tuning.The present application can fully fuse the key factors of working condition, reduce the cross-domain allocation deviation and knowledge interference when the heterogeneous knowledge scale is larger, improve the consistency and stability of generated content at the same time, and give priority to the better trade-off between reasoning accuracy and computing overhead.

Description

Technical Field

[0001] This invention relates to a hybrid expert efficient fine-tuning routing method for multimodal large-scale models of coal mine equipment operation and maintenance, belonging to the field of coal mine equipment operation and maintenance technology. Background Technology

[0002] Coal mine equipment operation and maintenance scenarios are characterized by diverse equipment types, rapid changes in operating conditions, and large-scale heterogeneous data from multiple sources. Maintenance personnel typically need to refer to maintenance manuals, historical work orders, inspection records, sensor trends, and safety procedures simultaneously during fault diagnosis and handling. In recent years, large language models have demonstrated outstanding performance in knowledge-based question answering and reasoning, and have begun to be used to assist in maintenance knowledge retrieval, fault cause analysis, and the generation of handling suggestions. However, in practical applications, full-scale fine-tuning often places high demands on computing power and storage resources, and may also lead to catastrophic forgetting issues, affecting the model's original versatility and output stability.

[0003] To reduce the computational and storage costs of fine-tuning large models, low-rank adaptive (LoRA) technology is widely used. It achieves efficient parameter fine-tuning by freezing pre-trained weights and injecting a low-rank increment matrix. However, coal mine production systems are large and complex, encompassing multiple key subsystems such as hoisting, ventilation, compressed air, drainage, and transportation. These subsystems exhibit significant domain differences and knowledge heterogeneity, making it difficult for a single adapter to simultaneously ensure both the accuracy of knowledge representation and the consistency of reasoning within a limited parameter space.

[0004] To improve the ability to handle heterogeneous knowledge while maintaining efficient parameter fine-tuning, existing technologies propose a hybrid expert efficient fine-tuning architecture. This architecture freezes the main parameters of the base model and sets multiple LoRA adapters in parallel as multiple experts. It then uses a routing network to select and weightedly fuse input features, achieving sparse activation and dynamic computational allocation. However, in the multimodal input and inference processes of coal mine equipment operation and maintenance, existing hybrid expert efficient fine-tuning routing mechanisms still have limitations. Their routing decisions are usually based on the scoring results of input features, making it difficult to fully integrate key factors such as operating conditions, which can easily lead to expert selection bias and affect inference consistency. Meanwhile, in long text generation and multi-turn interaction scenarios, expert selection is prone to fluctuations and output instability. Finally, fixed Top-k sparse activation strategies lack adaptability to changes in task complexity, making it difficult to achieve a better trade-off between inference accuracy and computational cost. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a hybrid expert efficient fine-tuning routing method for multimodal large-scale models of coal mine equipment operation and maintenance. This method can fully integrate key factors of operating conditions, reduce cross-domain allocation deviations and knowledge interference when the scale of heterogeneous knowledge is large, improve the consistency and stability of generated content, and take into account a better trade-off between inference accuracy and computational overhead.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a hybrid expert-based efficient fine-tuning routing method for multimodal large-scale models of coal mine equipment operation and maintenance, comprising the following steps:

[0007] S1. Multi-source heterogeneous data acquisition for coal mine equipment;

[0008] S2, Multimodal data feature extraction and joint embedding;

[0009] S3, Hierarchical Routing and Expert Probability Calculation;

[0010] S4, Smoothing out of routing stability;

[0011] S5, Adaptive Top-k Sparse Expert Activation and Load Constraints;

[0012] S6, high-efficiency fine-tuning by hybrid experts.

[0013] Furthermore, the multi-source heterogeneous data of coal mine equipment in S1 includes unstructured text data sources, real-time collected mining equipment operation data, and equipment attribute data sources; their acquisition processes are as follows:

[0014] (1) Obtain historical fault description text and natural language query instructions through human-computer interaction terminal as unstructured text data source;

[0015] (2) Real-time data collection of mining equipment operation data is collected from sensors deployed around and inside the mining equipment according to a set sampling period, including the wire rope tension of the main shaft hoist, brake oil pressure, motor stator temperature and vibration acceleration of the ventilation fan, and motor current and belt misalignment data of the belt conveyor; the collected operating data is extracted by setting a time sliding window to obtain the current time. A continuous time-series data segment within a second;

[0016] (3) Retrieve the static attribute fields of the target equipment from the coal mine equipment asset management system, including equipment type and installation location information; use the preset natural language prompt template to reorganize the above discrete attribute fields into structured equipment description text as the equipment attribute data source.

[0017] Furthermore, the specific process of S2 is as follows:

[0018] S2.1 Input the acquired unstructured text data into the word embedding layer of the pre-trained language model to extract the text semantic feature vector. ;

[0019] S2.2. Input the acquired continuous time-series data segments into a Long Short-Term Memory (LSTM) network for feature extraction, mapping the high-dimensional time-series signal into a fixed-length operating condition feature vector. ;

[0020] S2.3. Input the constructed structured device description text into the text encoder of the pre-trained language model for feature extraction to obtain the device attribute vector. ;

[0021] S2.4, Transfer the text semantic feature vector Operating condition feature vector With device attribute vector Multimodal alignment and nonlinear fusion are performed, and the input hidden state of the router is calculated according to formula (1). :

[0022] ;

[0023] in, and These are the weight matrix and bias vector of the multimodal fusion layer, respectively. This represents a vector concatenation operation. This is the activation function.

[0024] Furthermore, the specific process of S3 is as follows:

[0025] S3.1, Define the coal mine production system A cluster of subsystem domains, covering the lifting system, ventilation system, compressed air system, drainage system, and transportation system, will use the multimodal hidden states calculated by S2. The input is fed into a first-level domain routing network, which contains a trainable weight matrix. By performing matrix multiplication and Softmax normalization, the probability distribution of input features belonging to each subsystem domain cluster is calculated. :

[0026] ;

[0027] in, The dimension is , Hidden state The feature dimension is used to map the output to a probability distribution that sums to 1;

[0028] S3.2 For each subsystem domain cluster selected by the primary route Within this domain cluster, a set of expert adapters for subdivided fault handling is further deployed to calculate the first fault within this domain. Conditional activation probability of an expert adapter :

[0029] ;

[0030] in, For the first The second-level subdivision routing weight matrix corresponding to each domain cluster;

[0031] S3.3 Multiply the first-level domain routing probability by the second-level conditional probability to obtain the second-level... The original activation probability of each expert adapter in the global scope :

[0032]

[0033] Furthermore, the specific process of S4 is as follows:

[0034] S4.1 For the continuous token sequence generated in the operation and maintenance log generation task or the multi-round dialogue of fault diagnosis, maintain a historical routing state cache to record the expert selection probability distribution of the previous time step;

[0035] S4.2, at the current time step Obtain the original activation probability calculated at the current time. and the stable route probability of the previous time step The two values ​​are weighted and summed to obtain the corrected stable route probability at the current time. :

[0036]

[0037] in, The momentum smoothing coefficient is used to control the retention weight of historical routing states. When the model maintains the current expert combination, it tends to generate long texts that are stylistically consistent and logically coherent; when In this case, the model is more sensitive to semantic changes in the current input.

[0038] Furthermore, the specific process of S5 is as follows:

[0039] S5.1 Stable route probability distribution for the output Perform an evaluation and calculate the routing entropy. As a metric for measuring the complexity of the current operating conditions:

[0040] (6);

[0041] S5.2 Dynamically determine the number of experts that need to be activated. The judgment rule is: preset entropy threshold. and probability interval threshold If the calculated routing entropy Greater than the entropy threshold Or the probability difference between the top two experts in the ranking. Less than the probability interval threshold If the current operating condition is determined to be complex or on the fault boundary, the number of activated experts is increased to [number missing]. Otherwise, maintain the activation count. ;

[0042] S5.3. Introduce an online load penalty term to correct the routing probability, in order to avoid computational congestion of hotspot experts during inference: Obtain experts Cumulative load within the current inference window And the expert's preset capacity limit Combined with load balancing penalty coefficient Calculate the final selection score :

[0043]

[0044] According to the revised score Select the top rankings The experts as the current activation expert set .

[0045] Furthermore, the specific process of S6 is as follows:

[0046] S6.1, Maintain the weight matrix of the base large model It is in a frozen state, among which This represents the input latent vector obtained in step S2.4. Dimensions This represents the dimension of the model's output vector, for the set of activation experts output in step S5. Each expert in Construct a pair of trainable low-rank decomposition matrices , where the rank value ;

[0047] S6.2, The implicit vector obtained in step S2.4 Parallel input is applied to the base path and the activated LoRA expert path, and the Softmax function is used to select the path. Score of each expert Normalization yields the gating weights Calculate the predicted output of the hybrid expert layer :

[0048] (8);

[0049] In the formula, This is the scaling factor;

[0050] S6.3 Obtaining Standard Labels for Training Samples For example, standard fault diagnosis reports, etc., calculate the total loss function. :

[0051] ;

[0052] in, The cross-entropy loss function is used to measure prediction accuracy for the generation task. To compensate for load balancing losses and prevent route collapse, where The total number of experts, For the first The frequency of selection by an expert within a batch. This represents the expert's average routing probability. This is the balance coefficient;

[0053] S6.4 Minimize the total loss obtained in step S6.3. To achieve the goal, the gradient is calculated layer by layer from the output layer to the input layer using the chain rule:

[0054] For the LoRA expert path, calculate the loss function for activating the expert. low-rank matrix and partial derivatives and This enables targeted updates of expert parameters;

[0055] For the base model path, due to the weight matrix The gradient is frozen and is truncated here, so it is not calculated or updated.

[0056] For gated networks, calculate the loss function with respect to the gate weights. The gradient is then fed back to the weight matrix of the hierarchical router. and ;;

[0057] After the gradients converge through the hybrid expert layer, they continue to backpropagate to the input hidden state. and through the activation function Calculate the parameters of the multimodal fusion layer and The partial derivatives;

[0058] Finally, the Adamw optimizer is used to update the following set of trainable parameters: multimodal fusion layer parameters. , Hierarchical router weight matrix and And the low-rank matrices of each expert Base weight matrix It remains unchanged throughout the entire process.

[0059] This invention improves upon the traditional approach of relying primarily on textual semantics for routing decisions by employing multi-source heterogeneous data acquisition from coal mine equipment, multi-modal data feature extraction and joint embedding, hierarchical routing and expert probability calculation, routing stability smoothing, adaptive Top-k sparse expert activation and load constraints, and efficient fine-tuning of hybrid experts. It constructs a multi-modal feature representation that integrates working condition time-series features, equipment attribute embedding, and textual semantics, and adopts a coarse-to-fine two-layer hierarchical routing strategy. This makes expert scheduling more aligned with the organizational structure and professional division of labor in coal mine operation and maintenance tasks, thereby reducing cross-domain allocation bias and knowledge interference when dealing with large amounts of heterogeneous knowledge. Addressing the issue of fluctuating expert selection during long-text generation and multi-round diagnostic interactions, a routing momentum smoothing mechanism is introduced to reduce output inconsistencies caused by frequent expert switching, improving the consistency and stability of generated content. Furthermore, this invention combines routing entropy and online load constraints to design an adaptive Top-k activation strategy, enabling the model to dynamically adjust the number and weight allocation of activated experts based on task uncertainty and real-time load. This balances diagnostic accuracy with computational overhead, mitigating the risk of expert congestion during the inference phase. Attached Figure Description

[0060] Figure 1 This is a flowchart of the present invention;

[0061] Figure 2 This is a schematic diagram of the hybrid expert high-efficiency fine-tuning architecture in this invention. Detailed Implementation

[0062] The invention will now be further described with reference to the accompanying drawings.

[0063] like Figure 1 As shown, a hybrid expert-based efficient fine-tuning routing method for a multimodal large-scale model of coal mine equipment operation and maintenance includes the following steps:

[0064] S1. Multi-source heterogeneous data acquisition for coal mine equipment;

[0065] S2, Multimodal data feature extraction and joint embedding;

[0066] S3, Hierarchical Routing and Expert Probability Calculation;

[0067] S4, Smoothing out of routing stability;

[0068] S5, Adaptive Top-k Sparse Expert Activation and Load Constraints;

[0069] S6, high-efficiency fine-tuning by hybrid experts.

[0070] Example: The multi-source heterogeneous data of coal mine equipment in S1 includes unstructured text data sources, real-time collected mining equipment operation data, and equipment attribute data sources; the acquisition processes are as follows:

[0071] (1) Obtain historical fault description text and natural language query instructions through human-computer interaction terminal as unstructured text data source;

[0072] (2) Real-time data collection of mining equipment operation data is collected from sensors deployed around and inside the mining equipment according to a set sampling period, including the wire rope tension of the main shaft hoist, brake oil pressure, motor stator temperature and vibration acceleration of the ventilation fan, and motor current and belt misalignment data of the belt conveyor; the collected operating data is extracted by setting a time sliding window to obtain the current time. A continuous time-series data segment within a second;

[0073] (3) Retrieve the static attribute fields of the target equipment from the coal mine equipment asset management system, including equipment type and installation location information; use the preset natural language prompt template to reorganize the above discrete attribute fields into structured equipment description text as the equipment attribute data source.

[0074] As a preferred implementation method, the specific process of S2 is as follows:

[0075] S2.1 Input the acquired unstructured text data into the word embedding layer of the pre-trained language model to extract the text semantic feature vector. ;

[0076] S2.2. Input the acquired continuous time-series data segments into a Long Short-Term Memory (LSTM) network for feature extraction, mapping the high-dimensional time-series signal into a fixed-length operating condition feature vector. ;

[0077] S2.3. Input the constructed structured device description text into the text encoder of the pre-trained language model for feature extraction to obtain the device attribute vector. ;

[0078] S2.4, Transfer the text semantic feature vector Operating condition feature vector With device attribute vector Multimodal alignment and nonlinear fusion are performed, and the input hidden state of the router is calculated according to formula (1). :

[0079] ;

[0080] in, and These are the weight matrix and bias vector of the multimodal fusion layer, respectively. This represents a vector concatenation operation. This is the activation function.

[0081] As a preferred embodiment, the specific process of S3 is as follows:

[0082] S3.1, Define the coal mine production system A cluster of subsystem domains, covering the lifting system, ventilation system, compressed air system, drainage system, and transportation system, will use the multimodal hidden states calculated by S2. The input is fed into a first-level domain routing network, which contains a trainable weight matrix. By performing matrix multiplication and Softmax normalization, the probability distribution of input features belonging to each subsystem domain cluster is calculated. :

[0083] ;

[0084] in, The dimension is , Hidden state The feature dimension is used to map the output to a probability distribution that sums to 1;

[0085] S3.2 For each subsystem domain cluster selected by the primary route Within this domain cluster, a set of expert adapters for subdivided fault handling is further deployed to calculate the first fault within this domain. Conditional activation probability of an expert adapter :

[0086] ;

[0087] in, For the first The second-level subdivision routing weight matrix corresponding to each domain cluster;

[0088] S3.3 Multiply the first-level domain routing probability by the second-level conditional probability to obtain the second-level... The original activation probability of each expert adapter in the global scope :

[0089]

[0090] As a preferred embodiment, the specific process of S4 is as follows:

[0091] S4.1 For the continuous token sequence generated in the operation and maintenance log generation task or the multi-round dialogue of fault diagnosis, maintain a historical routing state cache to record the expert selection probability distribution of the previous time step;

[0092] S4.2, at the current time step Obtain the original activation probability calculated at the current time. and the stable route probability of the previous time step The two values ​​are weighted and summed to obtain the corrected stable route probability at the current time. :

[0093]

[0094] in, The momentum smoothing coefficient is used to control the retention weight of historical routing states. When the model maintains the current expert combination, it tends to generate long texts that are stylistically consistent and logically coherent; when In this case, the model is more sensitive to semantic changes in the current input.

[0095] For example: allocate a buffer in memory to store the previous moment. The expert probability distribution vector. For the current time step. In this embodiment, the generated continuous fault diagnosis description sequence utilizes a preset momentum coefficient. The original probability calculated at the current moment The stable probability compared to the previous moment Based on formula (5), a weighted smoothing iteration is performed to obtain the corrected stable route probability. .

[0096] As a preferred embodiment, the specific process of S5 is as follows:

[0097] S5.1 Stable route probability distribution for the output Perform an evaluation and calculate the routing entropy. As a metric for measuring the complexity of the current operating conditions:

[0098] (6);

[0099] S5.2 Dynamically determine the number of experts that need to be activated. The judgment rule is: preset entropy threshold. and probability interval threshold If the calculated routing entropy Greater than the entropy threshold Or the probability difference between the top two experts in the ranking. Less than the probability interval threshold If the current operating condition is determined to be complex or on the fault boundary, the number of activated experts is increased to [number missing]. Otherwise, maintain the activation count. ;

[0100] S5.3. Introduce an online load penalty term to correct the routing probability, in order to avoid computational congestion of hotspot experts during inference: Obtain experts Cumulative load within the current inference window And the expert's preset capacity limit Combined with load balancing penalty coefficient Calculate the final selection score :

[0101]

[0102] According to the revised score Select the top rankings The experts as the current activation expert set .

[0103] In this embodiment, the stable routing probability distribution is based on the output. First, calculate the information entropy of the current probability distribution in real time. If a conflict is detected between the hoist operating data and the text description, causing the entropy value to exceed a preset threshold, The system determines that the working conditions are complex and automatically increases the number of activated experts from the default value. Dynamically increase to To facilitate collaborative consultations with experts from more fields, the system prioritizes certain areas; conversely, for simple, explicit state queries, it maintains a minimum number of activations to conserve computational resources. Simultaneously, to prevent computational hotspots during inference, the system continuously monitors the cumulative load of each expert within the sliding window of their most recent 100 requests. And combined with load balancing penalty coefficient (In this embodiment, the value is set to 0.1). The routing scores of high-load experts are penalized and recalibrated to encourage computational traffic to be reasonably distributed to idle experts. Finally, the top-ranked experts are selected based on the corrected scores. The expert indexes constitute the current active set. .

[0104] As a preferred implementation method, such as Figure 2 As shown, the specific process of S6 is as follows:

[0105] S6.1, Maintain the weight matrix of the base large model It is in a frozen state, among which This represents the input latent vector obtained in step S2.4. Dimensions This represents the dimension of the model's output vector, and the set of activation experts for the S5 output. Each expert in Construct a pair of trainable low-rank decomposition matrices , where the rank value ;

[0106] S6.2, The implicit vector obtained in step S2.4 Parallel input is applied to the base path and the activated LoRA expert path, and the Softmax function is used to select the path. Score of each expert Normalization yields the gating weights Calculate the predicted output of the hybrid expert layer : (8);

[0107] In the formula, This is the scaling factor;

[0108] S6.3 Obtaining Standard Labels for Training Samples For example, standard fault diagnosis reports, etc., calculate the total loss function. :

[0109] ;

[0110] in, The cross-entropy loss function is used to measure prediction accuracy for the generation task. To compensate for load balancing losses and prevent route collapse, where The total number of experts, For the first The frequency of selection by an expert within a batch. This represents the expert's average routing probability. This is the balance coefficient;

[0111] S6.4 Minimize the total loss obtained in step S6.3. To achieve the goal, the gradient is calculated layer by layer from the output layer to the input layer using the chain rule:

[0112] For the LoRA expert path, calculate the loss function for activating the expert. low-rank matrix and partial derivatives and This enables targeted updates of expert parameters;

[0113] For the base model path, due to the weight matrix The gradient is frozen and is truncated here, so it is not calculated or updated.

[0114] For gated networks, calculate the loss function with respect to the gate weights. The gradient is then fed back to the weight matrix of the hierarchical router. and ;

[0115] After the gradients converge through the hybrid expert layer, they continue to backpropagate to the input hidden state. and through the activation function Calculate the parameters of the multimodal fusion layer and The partial derivatives;

[0116] Finally, the Adamw optimizer is used to update the following set of trainable parameters: multimodal fusion layer parameters. , Hierarchical router weight matrix and And the low-rank matrices of each expert Base weight matrix It remains unchanged throughout the entire process.

Claims

1. A hybrid expert-based efficient fine-tuning routing method for multimodal large-scale models of coal mine equipment operation and maintenance, characterized in that, Includes the following steps: S1. Multi-source heterogeneous data acquisition for coal mine equipment; S2, Multimodal data feature extraction and joint embedding; S3, Hierarchical Routing and Expert Probability Calculation; S4, Smoothing out of routing stability; S5, Adaptive Top-k Sparse Expert Activation and Load Constraints; S6, Hybrid Expert Efficient Fine-Tuning; The specific process of S2 is as follows: S2.1 Input the acquired unstructured text data into the word embedding layer of the pre-trained language model to extract the text semantic feature vector. ; S2.

2. Input the acquired continuous time-series data segments into a Long Short-Term Memory (LSTM) network for feature extraction, mapping the high-dimensional time-series signal into a fixed-length operating condition feature vector. ; S2.

3. Input the constructed structured device description text into the text encoder of the pre-trained language model for feature extraction to obtain the device attribute vector. ; S2.4, Transfer the text semantic feature vector Operating condition feature vector With device attribute vector Multimodal alignment and nonlinear fusion are performed, and the input hidden state of the router is calculated according to formula (1). : ； in, and These are the weight matrix and bias vector of the multimodal fusion layer, respectively. This represents the concatenation of the text semantic feature vector x, the operating condition feature vector s, and the equipment attribute vector t. For activation functions; The specific process of S3 is as follows: S3.1, Define the coal mine production system A cluster of subsystem domains, covering the lifting system, ventilation system, compressed air system, drainage system, and transportation system, will use the multimodal hidden states calculated by S2. The input is fed into a first-level domain routing network, which contains a trainable weight matrix. By performing matrix multiplication and Softmax normalization, the probability distribution of the input features belonging to each subsystem domain cluster is calculated according to formula (2). : ； in, The dimension is , Hidden state The feature dimension is used to map the output to a probability distribution that sums to 1; S3.2 For each subsystem domain cluster selected by the primary route Within this domain cluster, a set of expert adapters for subdivided fault handling are further deployed, and the fault handling is calculated according to formula (3). Conditional activation probability of an expert adapter : ； in, For the first The second-level subdivision routing weight matrix corresponding to each domain cluster; S3.3 Multiply the first-level domain routing probability by the second-level conditional probability to obtain the second-level... The original activation probability of each expert adapter in the global scope : （4）； The specific process of S4 is as follows: S4.1 For the continuous token sequence generated in the operation and maintenance log generation task or the multi-round dialogue of fault diagnosis, maintain a historical routing state cache to record the expert selection probability distribution of the previous time step; S4.2, at the current time step Obtain the original activation probability calculated at the current time. and the stable route probability of the previous time step The two values ​​are weighted and summed to obtain the corrected stable route probability at the current time. : （5）； in, The momentum smoothing coefficient is used to control the retention weight of historical routing states. When the model maintains the current expert combination, it tends to generate long texts that are stylistically consistent and logically coherent; when In this case, the model is more sensitive to semantic changes in the current input.

2. The hybrid expert efficient fine-tuning routing method for multimodal large-scale models of coal mine equipment operation and maintenance as described in claim 1, is characterized in that, The multi-source heterogeneous data of coal mine equipment in S1 includes unstructured text data sources, real-time collected mining equipment operation data, and equipment attribute data sources; the acquisition processes are as follows: (1) Obtain historical fault description text and natural language query instructions through human-computer interaction terminal as unstructured text data source; (2) Real-time data collection of mining equipment operation data is collected from sensors deployed around and inside the mining equipment according to a set sampling period, including the wire rope tension of the main shaft hoist, brake oil pressure, motor stator temperature and vibration acceleration of the ventilation fan, and motor current and belt misalignment data of the belt conveyor; the collected operating data is extracted by setting a time sliding window to obtain the current time. A continuous time-series data segment within a second; (3) Retrieve the static attribute fields of the target equipment from the coal mine equipment asset management system, including equipment type and installation location information; use the preset natural language prompt template to reorganize the above discrete attribute fields into structured equipment description text as the equipment attribute data source.

3. The hybrid expert efficient fine-tuning routing method for multimodal large-scale coal mine equipment operation and maintenance as described in claim 1, is characterized in that, The specific process of S5 is as follows: S5.1 Stable route probability distribution for the output Perform an evaluation and calculate the routing entropy. As a metric for measuring the complexity of the current operating conditions: （6）； S5.2 Dynamically determine the number of experts that need to be activated. The judgment rule is: preset entropy threshold. and probability interval threshold If the calculated routing entropy Greater than the entropy threshold Or the probability difference between the top two experts in the ranking. Less than the probability interval threshold If the current operating condition is determined to be complex or on the fault boundary, the number of activated experts is increased to [number missing]. ; Otherwise, maintain the activation count. ; S5.

3. Introduce an online load penalty term to correct the routing probability, in order to avoid computational congestion of hotspot experts during inference: Obtain experts Cumulative load within the current inference window And the expert's preset capacity limit Combined with load balancing penalty coefficient Calculate the final selection score : （7）； According to the revised score Select the top rankings The experts as the current activation expert set .

4. The hybrid expert efficient fine-tuning routing method for multimodal large-scale coal mine equipment operation and maintenance as described in claim 3, is characterized in that, The specific process of S6 is as follows: S6.1, Maintain the weight matrix of the base large model It is in a frozen state, among which, This represents the input latent vector obtained in step S2.

4. Dimensions This represents the dimension of the model's output vector, for the set of activation experts output in step S5. Each expert in Construct a pair of trainable low-rank decomposition matrices , where the rank value ; S6.2, The implicit vector obtained in step S2.4 Parallel input is applied to the base path and the activated LoRA expert path, and the Softmax function is used to select the path. Score of each expert Normalization yields the gating weights Calculate the predicted output of the hybrid expert layer : ； In the formula, This is the scaling factor; S6.3 Obtaining Standard Labels for Training Samples Calculate the total loss function : ; in, The cross-entropy loss function is used to measure prediction accuracy for the generation task. To compensate for load balancing losses and prevent route collapse, where The total number of experts, For the first The frequency of selection by an expert within a batch. This represents the expert's average routing probability. This is the balance coefficient; S6.4 Minimize the total loss obtained in step S6.

3. To achieve the goal, the gradient is calculated layer by layer from the output layer to the input layer using the chain rule: For the LoRA expert path, calculate the loss function for activating the expert. low-rank matrix and partial derivatives and This enables targeted updates of expert parameters; For the base model path, due to the weight matrix The gradient is frozen and is truncated here, so it is not calculated or updated. For gated networks, calculate the loss function with respect to the gate weights. The gradient is then fed back to the weight matrix of the hierarchical router. and ; After the gradients converge through the hybrid expert layer, they continue to backpropagate to the input hidden state. and through the activation function Calculate the parameters of the multimodal fusion layer and The partial derivatives; Finally, the Adamw optimizer is used to update the following set of trainable parameters: multimodal fusion layer parameters. , Hierarchical router weight matrix and And the low-rank matrices of each expert Base weight matrix It remains unchanged throughout the entire process.

Images