A power coal industry large model training optimization method based on knowledge distillation

Abstract

This invention discloses a method for training and optimizing a large-scale model in the thermal coal industry based on knowledge distillation, belonging to the field of large-scale model training and optimization technology. The method includes the following steps: collecting and preprocessing multimodal industry data to obtain a data matrix; inputting the data into a teacher's visual Transformer network to generate a sparse probability label matrix; constructing an improved TinyViT student network; embedding multiple optimization units in self-attention blocks; completing weight initialization; performing pre-training distillation to generate a composite distillation signal; inputting the signal into the student network for fine-tuning training to obtain a set of fine-tuned student weights; performing dynamic weighted training on hard samples; optimizing the training effect on long-tailed samples; and obtaining a lightweight model through quantized perceptual training and channel pruning. The model is then deployed to edge computing nodes for real-time inference. This invention improves the training efficiency of large-scale models in the thermal coal industry while optimizing the model's inference speed and accuracy, and can be effectively deployed in edge computing environments.

Description

Technical Field

[0001] This invention relates to the field of large model training and optimization technology, and in particular to a method for training and optimizing a large model in the thermal coal industry based on knowledge distillation. Background Technology

[0002] In the current thermal coal industry, with the rapid development of artificial intelligence and deep learning technologies, more and more applications are relying on big data-based models to optimize decision-making and improve efficiency. These large models typically need to process complex multimodal data, such as images, time-series data, and text, especially in areas like coal quality monitoring, transportation flow prediction, and energy consumption, where artificial intelligence has been widely applied. Existing deep learning models, particularly large models based on images and time-series data, often face bottlenecks in computing and storage resources, making it difficult to deploy them on edge devices.

[0003] Currently, many models are trained using traditional deep neural networks. These models typically have a large number of parameters and complex computational requirements. Although these models can achieve good performance when there is sufficient training data, traditional training methods often fail to meet the requirements of high efficiency and low resource consumption when faced with high-dimensional and large-scale multimodal data. This is especially true on resource-limited edge devices. Due to the lack of effective model compression and distillation strategies, existing models often cannot balance accuracy and inference speed, resulting in low application efficiency.

[0004] While existing knowledge distillation methods have made some progress in model compression and transfer learning, they still have some problems when dealing with industry-specific tasks. For example, traditional distillation methods often rely on hard label matching between teacher and student networks, lacking effective alignment of intermediate layer features and the integration of industry-specific knowledge. This results in the student network failing to fully learn the deep feature representations of the teacher network's intermediate layers during transfer learning. Furthermore, the distillation loss functions in existing methods often fail to adequately consider the training problem of long-tailed samples, leading to poor learning performance on these samples and affecting the model's generalization ability.

[0005] Therefore, how to provide a training and optimization method for a large-scale model of the thermal coal industry based on knowledge distillation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a training optimization method for a large-scale model in the thermal coal industry based on knowledge distillation. This invention fully utilizes knowledge distillation, deep learning, and multimodal data processing technologies, combined with industry characteristics such as coal quality and transportation flow, to optimize the training effect of the student network on long-tailed samples. By introducing feature preservation loss and dynamic weighted training of hard samples, this invention has efficient model compression capabilities, excellent inference performance, and strong industry adaptability, and can improve the inference efficiency and accuracy on edge computing devices.

[0007] A method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to an embodiment of the present invention includes the following steps: Image data, time-series data, and text data from the thermal coal industry were collected and preprocessed. Normalization, time alignment, and feature stitching were performed to obtain a multimodal industry data matrix. A multimodal industry data matrix is ​​input into a teacher vision Transformer network to generate a sparse probability label matrix. An improved TinyViT student network is constructed, and an energy regularization gating unit, a long-tail sample attention scheduling unit, and a quantization-friendly channel compression unit are set in each self-attention block of the improved TinyViT student network to form a student network structure. The weights of the TinyViT base layer, the energy regularization gating unit, the long-tail sample attention scheduling unit, and the quantization-friendly channel compression unit in the student network structure are initialized to obtain the initialized improved TinyViT student network. The multimodal industry data matrix and sparse probability label matrix are simultaneously input into the improved TinyViT student network for pre-training distillation, and the pre-distilled student weight set is output. The soft labels, intermediate layer features and industry energy gradients output by the teacher visual Transformer network are extracted and combined to generate a composite distillation signal. The composite distillation signal is input into the improved TinyViT student network for industry fine-tuning training to obtain the fine-tuned student weight set. Based on the difference between the fine-tuned student weight set and the output of the visual Transformer teacher network, dynamic weighted training of hard samples is performed to optimize the training effect of long-tail samples and generate the final optimized student weight set. Eight-bit quantization perception training and channel pruning are performed on the final optimized student weight set to obtain a lightweight large model of the thermal coal industry, which is then deployed to edge computing nodes for real-time inference.

[0008] Optionally, the step of inputting the multimodal industry data matrix into the teacher vision Transformer network and generating a sparse probability label matrix includes: Image embedding vectors from the multimodal industry data matrix are input into a convolutional neural network to extract local image features and unfold them into an image feature sequence. Temporal vectors from the multimodal industry data matrix are input into a one-dimensional convolutional network to extract time-series patterns and generate a temporal feature sequence. Text vectors from the multimodal industry data matrix are input into a pre-trained language model to extract semantic features and generate a text feature sequence. The image feature sequence, temporal feature sequence, and text feature sequence are concatenated sequentially to form a multimodal feature sequence, which is then input into a teacher vision Transformer network. Multi-head auto-analysis is then performed sequentially within the teacher vision Transformer network. Attention calculation, feedforward neural network calculation, layer normalization calculation, and residual connection operation are used to transform the multimodal feature sequence and output the transformed feature sequence. In the multi-head self-attention calculation process, a physical constraint matrix based on coal quality coupling and material flow conservation is introduced into each self-attention calculation to adjust the attention weights. The transformed feature sequence is input into the linear transformation unit and a softmax operation is performed to obtain the prediction probability vector. The probability components in the prediction probability vector that are less than a set threshold are cleared to generate a sparse probability label matrix, which is then written into the distillation signal buffer.

[0009] Optionally, the construction and weighting steps of the physical constraint matrix based on coal quality coupling and material flow conservation include: Obtain the coal quality attributes of each batch of thermal coal and construct a coal quality attribute matrix; The transportation information of different coal types in each time period is obtained, the flow distribution of each transportation route is calculated, and a logistics flow matrix is ​​formed. The coal quality attribute matrix is ​​combined with the logistics flow matrix, and the coupling relationship between coal quality and flow on each route is calculated based on the correlation between coal quality attributes and the physical constraints between transportation flow, thus generating an initial physical constraint matrix. The initial physical constraint matrix is ​​normalized, with the total physical constraint for each path being 1. The normalized physical constraint matrix is ​​then multiplied element-wise with the attention weight matrix in the self-attention calculation process to generate a physical constraint-guided attention weight matrix. In each encoder layer, the physical constraint-guided attention weight matrix replaces the original attention weight matrix for self-attention calculation, thereby achieving feature extraction based on the physical constraints of coal quality and logistics flow conservation.

[0010] Optionally, the improved TinyViT student network embeds the following structure in each self-attention block: The intermediate feature representations of the self-attention blocks corresponding to the multimodal industry data matrix in the improved TinyViT student network are input into the energy regularization gating unit. In the energy regularization gating unit, a channel energy vector is constructed based on the activation value of each channel in the training batch, the L2 norm of each channel energy is calculated, and a channel energy distribution map is generated. The normalized gating weights are calculated based on the channel energy distribution map, and the intermediate feature representations are weighted channel by channel according to the normalized gating weights to generate energy regularized feature representations. The energy-regularized feature representation is input into the long-tail sample attention scheduling unit. In the long-tail sample attention scheduling unit, the confidence distribution of the sparse probability label matrix corresponding to each training sample is statistically analyzed. Combined with the historical distillation loss of the teacher's visual Transformer network, a sample difficulty vector is constructed. The sample difficulty vector is projected onto the current attention mapping space and used as an attention bias term in the weighted calculation of the attention matrix to generate an attention map with difficulty-guided weights. Based on the attention map, the energy-regularized feature representation is weighted in the sample dimension to obtain the attention scheduling feature representation. The attention scheduling feature representation is input into the quantization-friendly channel compression unit. In the quantization-friendly channel compression unit, the channel stability index is estimated based on the mean and variance of the current channel dimension. The channels are ranked by importance, and the set of channels with a mean activation value greater than a preset mean threshold and a variance activation value less than a preset variance threshold is extracted. The channel set is then subjected to 8-bit fixed-point quantization and structured pruning, and the compressed quantized feature representation is output. The compressed quantized feature representation is then passed as the output of the current self-attention block to the next network layer, completing the feature transformation operation of the three-unit embedding.

[0011] Optionally, the pre-training distillation step includes: The multimodal industry data matrix and the corresponding sparse probability label matrix are used as input data and supervision signals, respectively, and fed into the improved TinyViT student network to perform supervised distillation training. In each training round, the soft label distillation loss is calculated based on the probability distribution distance between the current output of the improved TinyViT student network and the sparse probability label matrix, and backpropagated to update the student network parameters. The similarity between the feature representations generated by the improved TinyViT student network in the intermediate feature layer and the corresponding feature representations in the intermediate layer of the teacher visual Transformer network is evaluated, and a feature preservation loss is constructed and added to the total loss function. During training, the distillation error of each training sample is recorded, and a difficulty guidance factor is constructed based on the historical mean of the distillation error to dynamically weight the distillation loss of each sample. The learning rate and distillation weight factor are adjusted according to the cumulative training rounds and the preset stages. The convergence state is judged based on the loss decrease in adjacent training rounds. Several rounds of parameter updates are performed, and finally a pre-distilled student weight set is generated.

[0012] Optionally, the step of performing dynamic weighted training on difficult samples includes: calculating the distillation error of each training sample based on the difference between the fine-tuned student weight set and the teacher network output; generating a difficulty guidance factor based on the historical mean and volatility of the distillation error; dynamically weighting the distillation loss of each sample based on the difficulty guidance factor; and optimizing the training effect of the student network on long-tailed samples through weighted distillation loss.

[0013] Optionally, the step of performing octet quantization-based perceptual training and channel pruning on the final optimized student weight set includes: While maintaining the accuracy of the original model, the final optimized student weight set is subjected to eight-bit integer quantization perceptual training, the dynamic range of each layer weight and activation tensor is recorded, the corresponding quantization scaling factor and zero point value are generated, and a low bit width computation graph is constructed. Channel pruning is performed on the output channels of each convolutional layer to remove low-importance channels and simultaneously adjust the weight dimension and bias structure. After quantization and pruning, a compressed and lightweight large model of the thermal coal industry is exported, deployed to edge computing nodes, and loaded into the inference engine to support real-time inference and output of multimodal data of the thermal coal industry.

[0014] The beneficial effects of this invention are: (1) Improve the learning ability of long-tail samples. This invention solves the shortcomings of traditional methods in learning long-tail samples by introducing dynamic weighted training of difficult samples. By assigning higher weights to the more difficult long-tail samples, the student network can pay more attention to these difficult samples during the training process, thereby improving the prediction accuracy and generalization ability of the model on these samples.

[0015] (2) Optimize model compression and inference efficiency. By introducing eight-bit quantization for perceptual training and channel pruning, this invention achieves model lightweighting, enabling large models to reduce storage and computation requirements while ensuring accuracy. This allows the model to be deployed more efficiently on resource-constrained edge computing devices, providing fast real-time inference capabilities.

[0016] (3) Improve the adaptability and accuracy of industry-specific tasks. This invention combines industry energy gradient with the physical constraints of specific tasks, and effectively integrates industry domain knowledge into model training. In this way, the model shows higher accuracy and adaptability in multimodal data processing tasks in the thermal coal industry, and can better solve industry-specific problems such as coal quality monitoring and transportation flow prediction. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a knowledge-based distillation-based large-scale model training and optimization method for the thermal coal industry proposed in this invention. Figure 2 This is a structural diagram showing the alignment of intermediate layer features between the teacher vision Transformer network and the improved TinyViT student network proposed in this invention; Figure 3 This diagram illustrates the dynamic weighted training process for difficult samples in a knowledge-based distillation-based large-scale model training and optimization method for the thermal coal industry proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figure 1-3 A method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation includes the following steps: Image data, time-series data, and text data from the thermal coal industry were collected and preprocessed. Normalization, time alignment, and feature stitching were performed to obtain a multimodal industry data matrix. In this embodiment, image data of the thermal coal industry is collected, including overhead views of open-pit stockpiles, monitoring images of belt conveyors, and images of loading operations. The original images are uniformly scaled to 224×224 pixels, and data enhancement operations such as histogram equalization, random horizontal flipping, and ±10° random rotation are performed. The image data is standardized according to the channel mean of 0.485, 0.456, 0.406 and the variance of 0.229, 0.224, 0.225 to obtain the image embedding vector. Collect time-series data, including sensor data such as belt speed, washing volume, online calorific value, real-time sulfur content, motor current, and vibration acceleration. Perform linear interpolation on missing measurement points of each sensor. Remove outliers by using the three-times median absolute deviation method before interpolation. Then, perform z-score normalization on the data of each channel to obtain the time-series vector. Text data such as dispatch instructions, maintenance records and coal quality inspection reports are collected, uniformly encoded in UTF-8 and tabs and blank lines are removed, word segmentation is performed using a special dictionary for the thermal coal industry, stop words are removed, professional terms related to coal quality indicators and logistics are retained, the text data is truncated or padded to a fixed length of 512, a pre-trained word segmenter is used to map it into an index sequence, position encoding is added, and text vectors are obtained. Based on the timestamps in the time series data, the image embedding vectors and text vectors within the same time window are aligned; if a certain modality data is missing in the window, the effective features of the previous window are used to fill it in. Image embedding vectors, temporal vectors, and text vectors are concatenated in a fixed order to form a multimodal industry data matrix, which is then stored in a tensor buffer.

[0020] A multimodal industry data matrix is ​​input into a teacher vision Transformer network to generate a sparse probability label matrix. An improved TinyViT student network is constructed, and an energy regularization gating unit, a long-tail sample attention scheduling unit, and a quantization-friendly channel compression unit are set in each self-attention block of the improved TinyViT student network to form a student network structure. The weights of the TinyViT base layer, the energy regularization gating unit, the long-tail sample attention scheduling unit, and the quantization-friendly channel compression unit in the student network structure are initialized to obtain the initialized improved TinyViT student network. The multimodal industry data matrix and sparse probability label matrix are simultaneously input into the improved TinyViT student network for pre-training distillation, and the pre-distilled student weight set is output. The soft labels, intermediate layer features and industry energy gradients output by the teacher visual Transformer network are extracted and combined to generate a composite distillation signal. The composite distillation signal is input into the improved TinyViT student network for industry fine-tuning training to obtain the fine-tuned student weight set. In this embodiment, the teacher vision Transformer network performs forward propagation on the input multimodal industry data to generate soft labels for each training sample. The soft labels represent the class probability distribution assigned by the teacher network to each training sample. These labels contain the teacher network's deep understanding of the samples, especially when the class boundaries are unclear, the soft labels provide rich class information.

[0021] The intermediate layer features of the teacher network are extracted. These intermediate layer features reflect the high-level semantic information extracted by the teacher network when processing input data. Each layer of the teacher network extracts features at different levels from the data. These features serve as the knowledge representation of the teacher network and can guide the student network to learn more abstract features during the training process.

[0022] The teacher network also calculates industry-related energy gradient information. These gradients reflect the correlation between industry variables such as coal quality and transportation flow and the model output in specific application scenarios of the thermal coal industry. These industry energy gradients provide the student network with industry-specific knowledge, helping the student network to more accurately capture industry-specific patterns and rules when processing industry data.

[0023] The soft labels, intermediate layer features, and industry energy gradients output by the teacher network are combined to form a composite distillation signal. This composite signal contains the classification information, feature layer information, and industry knowledge of the teacher network. It is then input into the student network as a distillation signal. Through this composite signal, the student network can simultaneously learn the output behavior, feature representation, and industry domain knowledge of the teacher network.

[0024] The composite distillation signal is input into the pre-distillation student network for industry fine-tuning training. During the fine-tuning training process, the student network not only tries to imitate the behavior of the teacher network as much as possible, but also adjusts itself according to industry data and industry characteristics in order to achieve better performance in practical applications. After multiple rounds of training, the parameters of the student network will be updated, and finally the fine-tuned student weight set is obtained. This fine-tuned student network has stronger generalization ability and can better adapt to the actual scenario of the thermal coal industry.

[0025] Based on the difference between the fine-tuned student weight set and the output of the visual Transformer teacher network, dynamic weighted training of hard samples is performed to optimize the training effect of long-tail samples and generate the final optimized student weight set. Eight-bit quantization perception training and channel pruning are performed on the final optimized student weight set to obtain a lightweight large model of the thermal coal industry, which is then deployed to edge computing nodes for real-time inference.

[0026] In this embodiment, the step of inputting the multimodal industry data matrix into the teacher vision Transformer network and generating a sparse probability label matrix includes: Image embedding vectors from the multimodal industry data matrix are input into a convolutional neural network to extract local image features and expand them into image feature sequences; temporal vectors from the multimodal industry data matrix are input into a one-dimensional convolutional network to extract time series patterns and generate time series feature sequences; text vectors from the multimodal industry data matrix are input into a pre-trained language model to extract semantic features and generate text feature sequences. In this embodiment, the specific operations include: inputting the image embedding vectors in the multimodal industry data matrix into a convolutional neural network. The convolutional neural network consists of three layers of two-dimensional convolutional structures, with convolutional kernel sizes of 3×3, 3×3 and 1×1, and channel numbers of 64, 128 and 256, respectively. Each convolutional layer is followed by batch normalization and ReLU activation functions. After three convolutional layers, an image feature map with a size of 14×14×256 is generated. Then, a pixel-by-pixel flattening operation is performed to expand it into a 196×256-dimensional image feature sequence in column-major order. The time-series vectors in the multimodal industry data matrix are input into a one-dimensional convolutional network. The one-dimensional convolutional network adopts a causal convolutional structure with a sliding window length of 5, containing two convolutional layers with 64 and 128 convolutional channels. After each convolutional operation, a ReLU activation function is connected. The convolution result outputs 128-dimensional time-series channels, retaining the most recent 60 time steps, and constructing a 60×128-dimensional time-series feature sequence to represent the sensor change trend within a continuous one-minute range. The text vectors from the multimodal industry data matrix are input into a pre-trained language model based on the BERT structure. The input text sequence is 512 bytes long. After passing through a 12-layer Transformer encoder, the context-related embeddings corresponding to each token are extracted from the output of the last layer to form a 512×768-dimensional text feature sequence. The text feature sequence is used to encode coal type identification, task keywords, and quantitative descriptions in scheduling instructions, maintenance records, and coal quality reports. The image feature sequence, temporal feature sequence, and text feature sequence are concatenated in sequence to form a multimodal feature sequence. The multimodal feature sequence is then input into a teacher vision Transformer network. In the teacher vision Transformer network, multi-head self-attention calculation, feedforward neural network calculation, layer normalization calculation, and residual connection operations are performed sequentially to transform the multimodal feature sequence and output the transformed feature sequence. In this embodiment, when the teacher's visual Transformer network processes the input multimodal feature sequence, it specifically performs the following steps sequentially: multi-head self-attention calculation, feedforward neural network calculation, layer normalization calculation, and residual connection operation to achieve feature transformation. Specifically, this includes: The input multimodal feature sequence enters the multi-head self-attention computation module. In this module, the input feature sequence is first mapped into a query, key, and value matrix with dimensions [N, D], where N is the length of the input sequence and D is the feature dimension. In each head, the relevance score between the query and the key is calculated using a dot product. After Softmax normalization, this score is multiplied by the value matrix to obtain the output of each head. The outputs of all heads are concatenated column-wise, and finally, a linear transformation is performed to obtain the merged attention output matrix. This step helps the model capture long-range dependencies between different modalities.

[0027] The attention output is fed into the feedforward neural network computation module. The feedforward neural network consists of two fully connected layers. Each layer is followed by an activation function and a dropout layer to enhance nonlinearity and prevent overfitting. Specifically, the first fully connected layer expands the input feature dimension from D to 4D and then performs a nonlinear transformation through the ReLU activation function. The second fully connected layer compresses the feature dimension from 4D back to D. This process further enhances the expressive power of the features.

[0028] After the feedforward neural network completes its computation, layer normalization is performed. Layer normalization is used to standardize the output features of each layer by calculating the mean and standard deviation of each feature in that layer, subtracting the mean and dividing by the standard deviation to make the distribution of each feature close to zero mean and unit variance. Layer normalization helps to accelerate the training process, improve the stability of the model, and reduce gradient vanishing or gradient exploding problems during training.

[0029] After layer normalization calculation, residual connection operation is performed. The residual connection adds the layer normalized output to the input features to form the final output. This operation effectively avoids information loss in deep networks, allows information to flow between layers, avoids the gradient vanishing problem, and improves the convergence speed and performance of the model.

[0030] In each self-attention calculation, a physical constraint matrix based on coal quality coupling and logistics flow conservation is introduced to adjust the attention weights. The transformed feature sequence is input into the linear transformation unit and a Softmax operation is performed to obtain the prediction probability vector. The probability components in the prediction probability vector that are less than a set threshold are cleared to generate a sparse probability label matrix. The sparse probability label matrix is ​​written into the distillation signal buffer.

[0031] In this embodiment, the steps for constructing and weighting the physical constraint matrix based on coal quality coupling and material flow conservation include: Obtain the coal quality attributes of each batch of thermal coal and construct a coal quality attribute matrix; In this embodiment, the coal quality properties of each batch of thermal coal are first obtained. These properties include core indicators such as ash content, volatile matter, total sulfur, moisture, and calorific value. These indicators are obtained through data collection from sensor equipment in coal mines, transportation facilities, and power plants, laboratory test results, and online monitoring systems. The data collection process includes: Ash content: The ash content is determined by measuring the combustion process of raw coal samples, reflecting the mineral content in the coal. Volatile matter: The volatile substances lost by coal at a certain temperature are determined by heating method, indicating the volatile gas release characteristics of coal; Total sulfur: The sulfur content in coal is determined by chemical analysis and is a key indicator affecting coal combustion quality and environmental emissions. Moisture content: The moisture content in coal is determined by heating, which affects the calorific value and combustion efficiency of coal. Calorific value: The total energy release of coal is determined by calorimetry, which directly affects the combustion efficiency and economy of power plants.

[0032] The above coal quality data were obtained through on-site sensors, laboratory testing, and periodic sampling tests, and were summarized by batch. After data cleaning and outlier removal, each batch of coal quality data formed a batch coal quality attribute vector, in which each coal quality index corresponds to a numerical value.

[0033] The coal quality attribute values ​​of each batch are arranged in a fixed order to construct a coal quality attribute matrix. The rows of the matrix represent different batches of coal samples, and the columns represent different coal quality indicators. Each element of the matrix is ​​the value of a certain batch of coal samples on a specific coal quality indicator. The coal quality attribute matrix serves as the input for calculation and optimization, ensuring the integrity and accuracy of coal quality data and providing effective basic information on coal quality for model training.

[0034] The transportation information of different coal types in each time period is obtained, the flow distribution of each transportation route is calculated, and a logistics flow matrix is ​​formed. The coal quality attribute matrix is ​​combined with the logistics flow matrix, and the coupling relationship between coal quality and flow on each route is calculated based on the correlation between coal quality attributes and the physical constraints between transportation flow, thus generating an initial physical constraint matrix. In this embodiment, a physical constraint matrix is ​​calculated and constructed based on the correlation between coal quality attributes and the physical constraints between transportation flow. The physical constraints between coal quality attributes and transportation flow reflect the actual relationship between coal quality and logistics flow in the thermal coal industry, ensuring that the model follows the industry's business rules and physical constraints. The correlation between coal quality attributes refers to the inherent relationship between different coal quality indicators, for example: Calorific value and volatile matter: Generally, higher volatile matter content is positively correlated with higher calorific value, because volatile matter is the combustible gas released during the combustion of coal, and coal with high volatile matter content usually has a higher calorific value. The inverse relationship between moisture content and calorific value: High-moisture coal usually has a lower calorific value because the evaporation of moisture consumes some of the heat. Total sulfur and volatile matter: In some coal types, the total sulfur content and volatile matter content have a certain positive correlation, which is related to the gasification characteristics of coal. In order to quantify the correlation between these coal quality attributes, we first calculate the Pearson correlation coefficient between each coal quality attribute to obtain a coal quality attribute correlation matrix. Each element in the coal quality attribute correlation matrix represents the correlation strength between different coal quality indicators. The physical constraints of transportation flow refer to the flow conservation relationship during coal transportation. Since coal transportation usually follows certain routes, the flow along each transportation route should satisfy the following basic physical constraints: Flow conservation: The input flow of each transportation path should equal the output flow, especially in the transportation of coal from the mining site to the power plant or port, the flow should remain balanced at any time. The constraint relationship between coal quality and flow rate: Coal quality attributes are related to coal transportation flow rate. Coal with better quality is usually transported first and has a larger transportation flow rate, while coal with poor quality may have a smaller transportation flow rate due to cost and efficiency issues. Therefore, the relationship between the flow rate and coal quality indicators of each path in the transportation flow matrix needs to be established through physical constraints to ensure that the flow rate on each transportation path conforms to the actual logistics constraints. By fusing the coal quality attribute matrix and the transport flow matrix, and based on the correlation of coal quality attributes and the physical constraints of flow, we construct a physical constraint matrix. The elements in the physical constraint matrix represent the constraint relationships between different coal qualities and transport flows, including the following aspects: The flow distribution is corrected based on the correlation of coal quality to ensure that the transport flow of high volatile coal types is higher than that of low volatile coal types; Based on the principle of flow conservation, the flow of each transportation path is adjusted to make it conform to the balance of input and output flow; The physical constraint matrix is ​​used for weighted adjustments in the self-attention mechanism of the Transformer network to ensure that the model follows the physical relationship between coal quality and transportation flow during the learning process. This effectively incorporates industry knowledge and physical constraints into the model prediction, improving the accuracy and interpretability of the prediction.

[0035] The initial physical constraint matrix is ​​normalized, with the total physical constraint for each path being 1, to reflect the resource conservation constraints within the industry. The normalized physical constraint matrix is ​​then multiplied element-wise with the attention weight matrix in the self-attention calculation process to generate a physical constraint-guided attention weight matrix. In each encoder layer, the physical constraint-guided attention weight matrix replaces the original attention weight matrix for self-attention calculation, enabling feature extraction based on the physical constraints of coal quality and logistics flow conservation.

[0036] In this embodiment, the improved TinyViT student network embeds the following structure in each self-attention block: The intermediate feature representations of the self-attention blocks corresponding to the multimodal industry data matrix in the improved TinyViT student network are input into the energy regularization gating unit. In the energy regularization gating unit, a channel energy vector is constructed based on the activation value of each channel in the training batch, the L2 norm of each channel energy is calculated, and a channel energy distribution map is generated. The normalized gating weights are calculated based on the channel energy distribution map, and the intermediate feature representations are weighted channel by channel according to the normalized gating weights to generate energy regularized feature representations. The energy-regularized feature representation is input into the long-tail sample attention scheduling unit. In the long-tail sample attention scheduling unit, the confidence distribution of the sparse probability label matrix corresponding to each training sample is statistically analyzed. Combined with the historical distillation loss of the teacher's visual Transformer network, a sample difficulty vector is constructed. The sample difficulty vector is projected onto the current attention mapping space and used as an attention bias term in the weighted calculation of the attention matrix to generate an attention map with difficulty-guided weights. Based on the attention map, the energy-regularized feature representation is weighted in the sample dimension to obtain the attention scheduling feature representation. In this embodiment, the historical distillation loss is the KL divergence between the student model output and the teacher model output accumulated during the training process for each training sample; the loss is used to characterize the distillation difficulty of the sample. The greater the difficulty, the higher the distillation loss, which is given higher weight in the subsequent attention scheduling stage, guiding the student network to learn more fully the representation ability of such difficult samples in the teacher model.

[0037] The attention graph with difficulty-guided weights is constructed by fusing the sample history distillation loss to build a training difficulty bias vector based on standard self-attention calculation. This bias vector is added to the attention logits and a softmax operation is performed. The resulting weight matrix is ​​used to dynamically amplify the key feature location information in long-tail samples, thereby improving the student network's attention response strength to low-frequency samples.

[0038] The attention scheduling feature representation is input into the quantization-friendly channel compression unit. In the quantization-friendly channel compression unit, the channel stability index is estimated based on the mean and variance of the current channel dimension. The channels are ranked by importance, and a set of high-stability and high-activation channels is extracted. The channel set is then subjected to 8-bit fixed-point quantization and structured pruning, and the compressed quantized feature representation is output. The compressed quantized feature representation is then passed as the output of the current self-attention block to the next network layer, completing the feature transformation operation of the three-unit embedding.

[0039] In this embodiment, in the quantization-friendly channel compression unit, for the intermediate feature representation of each self-attention block in the student network, the set of activation values ​​of each channel on multiple training samples is extracted along the channel dimension, and the average value and fluctuation of the activation value of the channel are statistically analyzed; further, based on the average activation value and relative fluctuation of each channel, a channel scoring rule is constructed to simultaneously measure activation intensity and response stability.

[0040] In this embodiment, constructing a channel scoring rule refers to establishing a decision criterion for evaluating the retention value of a channel based on the activation characteristics of each channel in the training samples, taking into account the average level and fluctuation range of the activation value. Specifically, by setting weight factors, the preference for highly activated channels and the trust in stable channels are balanced, thereby achieving a score for the importance of different channels. The scoring rule serves as the basis for channel selection, ensuring that the retained channels are representative in terms of expressive power and stability, which helps to improve the performance of the model after quantization and compression.

[0041] In the scoring rules, different weights are assigned to activation intensity and volatility, and a comprehensive importance score is generated for each channel accordingly. The scores of all channels are sorted, and the top-scoring channels are extracted to form a target retention channel set. Quantitative perceptual training and channel pruning operations are performed on the target channel set to complete the structural compression and parameter simplification of the student network.

[0042] In this embodiment, the step of performing pre-training distillation includes: The multimodal industry data matrix and the corresponding sparse probability label matrix are used as input data and supervision signals, respectively, and fed into the improved TinyViT student network to perform supervised distillation training. In each training round, the soft label distillation loss is calculated based on the probability distribution distance between the current output of the improved TinyViT student network and the sparse probability label matrix, and backpropagated to update the student network parameters. In this embodiment, during pre-training distillation, the student network calculates the soft label distillation loss based on the probability distribution difference between the current output and the sparse probability label matrix in each training round, and uses the soft label distillation loss to backpropagate and update the parameters of the student network.

[0043] The student network's output generates a probability distribution through a Softmax layer, which is then compared with the sparse probability label matrix generated by the teacher network through the Softmax layer. The probability distribution output by the teacher network represents the teacher network's confidence in classifying each training sample. These probability values ​​are stored in the sparse probability label matrix. The goal of the student network is to adjust its output through training to be as close as possible to the teacher network's output.

[0044] In this process, distillation loss is usually measured by calculating the difference between the probability distribution of the student network output and the probability distribution of the teacher network output. Kullback-Leibler divergence or cross-entropy is usually used as the loss function. KL divergence measures the difference between the probability distributions of the student and teacher networks. The smaller the value, the smaller the difference between the two. The smaller the KL divergence value, the closer the student network's performance on the sample is to that of the teacher network.

[0045] After the distillation loss is calculated, the backpropagation process begins. The loss value is propagated to each layer of the network through the chain rule to calculate the gradient of each layer. These gradients indicate how the parameters of each layer need to be adjusted to reduce the gap between the current output and the target output. Through backpropagation, the student network will gradually adjust the weights so that the difference between the student network output and the teacher network output gradually decreases, thereby mimicking the performance of the teacher network.

[0046] Specifically, to improve the effectiveness of distillation training, a temperature factor can be introduced. The temperature factor is used to smooth the output probability distribution of the teacher network and reduce excessive bias in classification decisions. When the temperature is higher, the output probability distribution of the teacher network is smoother, which helps the student network learn richer category information and improves its ability to learn complex samples. Through temperature adjustment, the output of the Softmax layer becomes smoother, enabling the student network to learn the knowledge of the teacher network better.

[0047] Through the above steps, the student network continuously adjusts its parameters in each training round based on distillation loss and backpropagation, learning from the teacher network's knowledge, thereby achieving higher accuracy and better generalization ability during the optimization process.

[0048] The similarity between the feature representations generated by the improved TinyViT student network in the intermediate feature layer and the corresponding feature representations in the intermediate layer of the teacher visual Transformer network is evaluated, and a feature preservation loss is constructed and added to the total loss function. In this embodiment, the improved TinyViT student network performs similarity assessment with the intermediate layer feature representations of the teacher's visual Transformer network and calculates feature preservation loss to promote the student network to maintain similar feature representations to the teacher network during training. The student network performs forward propagation on each training sample to generate intermediate layer feature representations. These feature representations are generated through the student network's self-attention module and other computational layers, reflecting the student network's high-level semantic understanding of the sample. The teacher network also generates intermediate layer feature representations for the same training samples. These features are calculated by the teacher network through its self-attention module and intermediate layers, representing the teacher network's processing method for the sample.

[0049] The intermediate layer feature representations of the student network and the teacher network are compared to evaluate their similarity. This evaluation is achieved by calculating the similarity between the student and teacher networks in the intermediate layer features. Specifically, by measuring the similarity between the two feature representations, a similarity score is calculated. The student network can learn the high-level feature information extracted by the teacher network when processing samples, and better maintain feature consistency with the teacher network during training. This avoids excessive feature deviation in the student network during training, and helps the student network not only learn the output labels of the teacher network, but also effectively imitate at the feature level, thereby improving the model's performance and generalization ability.

[0050] During training, the distillation error of each training sample is recorded, and a difficulty guidance factor is constructed based on the historical mean of the distillation error to dynamically weight the distillation loss of each sample. In this embodiment, for each training sample, the distillation error is calculated. The distillation error represents the difference between the student network output and the teacher network output, and is usually measured by the distance between the probability distributions of the student network output and the teacher network output. Specifically, the distillation error is calculated based on the KL divergence between the student network output and the teacher network soft label. The magnitude of the distillation error reflects the learning difficulty of the student network when imitating the teacher network. When the distillation error is large, it indicates that there is a large gap between the student network and the teacher network, and the student network has not yet learned enough feature information on the sample.

[0051] Based on the distillation error of each training sample, a difficulty guidance factor is calculated. The difficulty guidance factor is dynamically adjusted according to the mean and fluctuation of the historical distillation error of each sample. Specifically, for each training sample, the distillation error of the sample during the training process is recorded first, and the historical mean of the sample's distillation error is calculated. The difficulty guidance factor of the sample is adjusted according to the deviation of the sample's distillation error from the historical mean. The more difficult the sample (i.e., the larger the deviation of the distillation error), the higher the guidance factor value. In this way, long-tail samples or samples with greater difficulty will receive greater weight.

[0052] The difficulty-guided factor is used to dynamically weight the distillation loss of each training sample. Specifically, the student network multiplies the distillation loss of each sample by the corresponding difficulty-guided factor, thereby giving more training attention to more difficult samples. This ensures that the student network can learn more on long-tail samples. In this way, the student network can concentrate its resources on learning more difficult samples, avoid the dominant influence of simple samples on training, and improve its ability to express long-tail samples and the overall performance of the model.

[0053] The learning rate and distillation weight factor are adjusted according to the cumulative training rounds and the preset stages. The convergence state is judged based on the loss decrease in adjacent training rounds. Several rounds of parameter updates are performed, and finally a pre-distilled student weight set is generated.

[0054] In this embodiment, the learning rate refers to the step size by which the student network adjusts along the gradient direction each time the parameters are updated. To ensure the stability and efficiency of the training process, the learning rate will gradually decay according to the progress of training. In the early stage of training, a larger learning rate is used to accelerate convergence. As training progresses, when the loss approaches the optimal value, the learning rate is gradually reduced to avoid over-updating and ensure convergence to the global optimum. The decay of the learning rate can be automatically adjusted through a preset decay strategy, such as exponential decay or cosine decay.

[0055] The distillation weight factor is a hyperparameter used to adjust the relative importance of distillation loss in the total loss. In the early stages of training, the distillation weight factor is usually high, prompting the student network to imitate the output of the teacher network as much as possible during the learning process. As training progresses and the student network gradually masters the knowledge of the teacher network, the distillation weight factor will gradually decrease to focus more on the student network's own learning task, such as classification or regression. By dynamically adjusting the distillation weight factor, the student network can find a suitable balance between learning the knowledge of the teacher network and learning its own task.

[0056] In this embodiment, the step of performing dynamic weighted training on difficult samples includes: calculating the distillation error of each training sample based on the difference between the fine-tuned student weight set and the teacher network output; generating a difficulty guidance factor based on the historical mean and volatility of the distillation error; and dynamically weighting the distillation loss of each sample according to the difficulty guidance factor. By weighting the distillation loss, the training effect of the student network on long-tailed samples is optimized.

[0057] In this embodiment, the steps of performing octet quantization-based perceptual training and channel pruning on the final optimized student weight set include: While maintaining the accuracy of the original model, the final optimized student weight set is subjected to eight-bit integer quantization perceptual training, the dynamic range of each layer weight and activation tensor is recorded, the corresponding quantization scaling factor and zero point value are generated, and a low bit width computation graph is constructed. Channel pruning is performed on the output channels of each convolutional layer to remove low-importance channels and simultaneously adjust the weight dimension and bias structure. After quantization and pruning, a compressed and lightweight large model of the thermal coal industry is exported, deployed to edge computing nodes, and loaded into the inference engine to support real-time inference and output of multimodal data of the thermal coal industry.

[0058] In this embodiment, eight-bit integer quantization-aware training is performed on the final optimized student network weight set to compress the model's storage requirements and improve inference efficiency. The specific steps of quantization-aware training are as follows: Dynamic range calculations are performed on the weight tensors and activation tensors of each network layer. By forward propagating batches of training data, the minimum and maximum values ​​of the weights and activations of each layer are recorded. These dynamic ranges reflect the response range of neurons in each layer under different inputs and play a key role. The calculated minimum and maximum values ​​are used in the quantization process to ensure that the values ​​of each layer can be effectively mapped to the quantized integer range.

[0059] Based on the dynamic range of the weights and activation tensors of each layer, corresponding quantization scaling factors and zero values ​​are generated. The quantization scaling factor is used to map the floating-point values ​​of each layer to the target integer range. For example, 8-bit quantization maps to the integer range of 0 to 255. The zero value is used to handle the mapping problem between negative numbers and zero, ensuring that negative values ​​can be correctly quantized to the corresponding integer range. These two parameters are crucial factors in the quantization process, determining the accuracy and performance of the network after quantization.

[0060] Based on the calculated quantization scaling factor and zero-point value, a low-bit-width computation graph is constructed for each layer. The low-bit-width computation graph refers to the use of a lower bit width, such as 8-bit integer representation, to replace the original 32-bit floating-point representation. Through quantization operations, both activation values ​​and weight values ​​are converted to integers, reducing storage and computational resource consumption. The constructed low-bit-width computation graph will be adjusted according to the quantized weights and activation tensors. In the inference process, integer operations are used instead of floating-point operations to improve computational efficiency.

[0061] By using 8-bit quantization for perceptual training and constructing a low-bit-width computation graph, the student network model is converted into a lightweight version, preparing for channel pruning operations and model deployment to edge computing nodes. Quantization operations reduce the model's storage space requirements, thereby improving the model's inference speed on edge devices by reducing computational resource consumption.

[0062] Example 1: To verify the feasibility of this invention in practice, it was applied to a real-world scenario of coal quality monitoring and transportation flow prediction in the thermal coal industry. This scenario involves multiple multimodal data sources, including coal quality testing data from coal mines, traffic flow information from transport vehicles, and related environmental data. Due to the high temporal and spatial correlation of these data, traditional deep learning methods face bottlenecks in processing power and computing resources. Therefore, the main problem we face is how to reduce the storage and computational overhead of the model while ensuring its accuracy, and at the same time improve the inference speed of the model in practical applications.

[0063] In this application scenario, we first collect multimodal data over different time periods, including coal quality data, transportation flow, and transportation routes. This data is recorded in real time using various sensors and data acquisition devices. Coal quality data mainly includes the coal's composition, calorific value, and moisture content. Transportation flow data includes the cargo load and transportation frequency at each transportation node. Route information covers the geographical information and transportation duration of the transportation routes. Through data preprocessing and time alignment, we obtain a standardized multimodal data matrix, providing a foundation for model training.

[0064] We train the student network using a knowledge distillation-based approach. The teacher network is obtained by training a large-scale deep vision Transformer model, which is responsible for processing these multimodal data and providing soft labels, feature representations, and energy gradients. Based on the soft labels output by the teacher network, we perform pre-training distillation on the student network. To optimize the performance of the student network, we also introduce a combination of intermediate layer feature alignment and industry energy gradients to ensure that the student network not only learns the final prediction of the teacher network, but also learns the teacher network's representations at the feature layer and industry-specific knowledge.

[0065] To further improve the efficiency of the model, we adopted a dynamic weighted training method for difficult samples during the training of the student network. By calculating the distillation error of each training sample, we constructed a difficulty guidance factor based on the historical mean of the distillation error. We gave higher training weights to long-tail samples, thereby ensuring that the student network can effectively learn those samples that are more difficult and account for a smaller proportion of the training data.

[0066] At the end of training, we successfully compressed the model into a lightweight version suitable for deployment on edge devices by using 8-bit quantization-aware training and channel pruning operations. This process reduces the model's storage requirements and computational overhead while maintaining the model's accuracy and generalization ability.

[0067] To verify the effectiveness of this invention, we compared it with traditional deep learning methods after applying the method. Table 1 shows the advantages of our method in multiple metrics. The following are comparative data on training time, model size, inference speed and prediction accuracy between the student network optimized by this invention and traditional deep learning methods.

[0068] Table 1: Comparison of Optimization Effects of Large-Scale Model in the Thermal Coal Industry As shown in Table 1, the method of this invention demonstrates optimization effects in several aspects compared to traditional deep learning methods. Regarding model size, the traditional deep learning method produces a model size of 350MB, while the optimized method of this invention significantly reduces the model size to 90MB, approximately one-quarter of the original. This indicates that the optimized model achieves substantial storage compression, making it suitable for deployment on edge devices with limited storage space.

[0069] Regarding training time, the traditional method takes 48 hours, while the optimized method of this invention reduces the training time to 24 hours, a reduction of about 50%. This shows that by optimizing the algorithm, this invention improves training efficiency, accelerates model iteration and application, and improves work efficiency, especially in scenarios where frequent model updates and optimizations are required.

[0070] In terms of inference time, traditional methods require 50ms to infer each sample, while the optimized method only requires 15ms, improving the inference speed by about three times. This is of great significance for real-time inference and large-scale data processing, especially in application scenarios that require fast response. This invention improves the inference efficiency of the model, enabling it to operate in real-time in actual production environments.

[0071] In terms of prediction accuracy, the traditional deep learning method achieves an accuracy of 85.2%, while the optimized method of this invention improves the accuracy to 88.7%, an increase of about 3.5 percentage points. This indicates that after knowledge distillation and long-tail sample weighted training, the model's performance on the prediction task has been improved, especially in the ability to identify complex samples and long-tail samples. The optimized model shows stronger generalization ability.

[0072] Distillation error is an indicator that measures the difference in output between the student network and the teacher network. The traditional method has a distillation error of 0.18, while the optimized method of this invention reduces the distillation error to 0.09, which proves that the optimized student network can learn the features and knowledge of the teacher network more accurately.

[0073] For long-tailed samples, the accuracy of traditional methods is 81.5%, while the accuracy of this invention is improved to 87.3%, an increase of nearly 6 percentage points. This improvement indicates that, through the hard sample dynamic weighted training in this invention, the student network exhibits stronger learning ability when dealing with long-tailed samples.

[0074] In summary, the optimization method of this invention not only improves the inference speed and prediction accuracy of the model, but also greatly reduces the model's storage requirements, optimizes the learning effect of long-tailed samples, and demonstrates performance improvement and application value.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation, characterized in that, Includes the following steps: Image data, time-series data, and text data from the thermal coal industry were collected and preprocessed. Normalization, time alignment, and feature stitching were performed to obtain a multimodal industry data matrix. The multimodal industry data matrix is ​​input into the teacher vision Transformer network to generate a sparse probability label matrix; an improved TinyViT student network is constructed, and an energy regularization gating unit, a long-tail sample attention scheduling unit, and a quantization-friendly channel compression unit are set in each self-attention block of the improved TinyViT student network to form a student network structure. The weights of the TinyViT base layer, energy regularization gating unit, long-tail sample attention scheduling unit, and quantization-friendly channel compression unit in the student network structure are initialized to obtain the initialized improved TinyViT student network. The multimodal industry data matrix and sparse probability label matrix are simultaneously input into the improved TinyViT student network for pre-training distillation, and the pre-distilled student weight set is output. The soft labels, intermediate layer features, and industry energy gradients output by the teacher visual Transformer network are extracted and combined to generate a composite distillation signal. The composite distillation signal is input into the improved TinyViT student network for industry fine-tuning training to obtain the fine-tuned student weight set. Based on the difference between the fine-tuned student weight set and the output of the visual Transformer teacher network, dynamic weighted training of hard samples is performed to optimize the training effect of long-tail samples and generate the final optimized student weight set. Eight-bit quantization perception training and channel pruning are performed on the final optimized student weight set to obtain a lightweight large model of the thermal coal industry, which is then deployed to edge computing nodes for real-time inference.

2. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 1, characterized in that, The steps of inputting the multimodal industry data matrix into the teacher vision Transformer network and generating a sparse probability label matrix include: Image embedding vectors from the multimodal industry data matrix are input into a convolutional neural network to extract local image features and unfold them into an image feature sequence. Temporal vectors from the multimodal industry data matrix are input into a one-dimensional convolutional network to extract time-series patterns and generate a temporal feature sequence. Text vectors from the multimodal industry data matrix are input into a pre-trained language model to extract semantic features and generate a text feature sequence. The image feature sequence, temporal feature sequence, and text feature sequence are concatenated sequentially to form a multimodal feature sequence, which is then input into a teacher vision Transformer network. The network sequentially performs multi-head self-attention computation, feedforward neural network computation, layer normalization computation, and residual connection operations to transform the multimodal feature sequence and output the transformed feature sequence. In the multi-head self-attention computation process, a physical constraint matrix based on coal quality coupling and logistics flow conservation is introduced into each self-attention computation to adjust the attention weights. The transformed feature sequence is input into the linear transformation unit and a Softmax operation is performed to obtain the prediction probability vector. The probability components in the prediction probability vector that are less than a set threshold are cleared to zero to generate a sparse probability label matrix.

3. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 2, characterized in that, The steps for constructing and weighting the physical constraint matrix based on coal quality coupling and logistics flow conservation include: Obtain the coal quality attributes of each batch of thermal coal and construct a coal quality attribute matrix; The transportation information of different coal types in each time period is obtained, the flow distribution of each transportation route is calculated, and a logistics flow matrix is ​​formed. The coal quality attribute matrix is ​​combined with the logistics flow matrix, and the coupling relationship between coal quality and flow on each route is calculated based on the correlation between coal quality attributes and the physical constraints between transportation flow, thus generating an initial physical constraint matrix. The initial physical constraint matrix is ​​normalized, with the total physical constraint for each path being 1. The normalized physical constraint matrix is ​​then multiplied element-wise with the attention weight matrix in the self-attention calculation process to generate a physical constraint-guided attention weight matrix. In each encoder layer, the physical constraint-guided attention weight matrix replaces the original attention weight matrix for self-attention calculation, thereby achieving feature extraction based on the physical constraints of coal quality and logistics flow conservation.

4. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 3, characterized in that, The improved TinyViT student network embeds the following structure in each self-attention block: The intermediate feature representations of the self-attention blocks corresponding to the multimodal industry data matrix in the improved TinyViT student network are input into the energy regularization gating unit. In the energy regularization gating unit, a channel energy vector is constructed based on the activation value of each channel in the training batch, the L2 norm of each channel energy is calculated, and a channel energy distribution map is generated. The normalized gating weights are calculated based on the channel energy distribution map, and the intermediate feature representations are weighted channel by channel according to the normalized gating weights to generate energy regularized feature representations. The energy-regularized feature representation is input into the long-tail sample attention scheduling unit. In the long-tail sample attention scheduling unit, the confidence distribution of the sparse probability label matrix corresponding to each training sample is statistically analyzed. Combined with the historical distillation loss of the teacher's visual Transformer network, a sample difficulty vector is constructed. The sample difficulty vector is projected onto the current attention mapping space and used as an attention bias term in the weighted calculation of the attention matrix to generate an attention map with difficulty-guided weights. Based on the attention map, the energy-regularized feature representation is weighted in the sample dimension to obtain the attention scheduling feature representation. The attention scheduling feature representation is input into the quantization-friendly channel compression unit. In the quantization-friendly channel compression unit, the channel stability index is estimated based on the mean and variance of the current channel dimension. The channels are ranked by importance, and the set of channels with the mean activation value greater than the preset mean threshold and the variance activation value less than the preset variance threshold is extracted. The channel set is then subjected to 8-bit fixed-point quantization and structured pruning, and the compressed quantized feature representation is output. The compressed quantized feature representation is passed as the output of the current self-attention block to the next network layer, completing the feature transformation operation of the embedded three units.

5. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 4, characterized in that, The pre-training distillation step includes: The multimodal industry data matrix and the corresponding sparse probability label matrix are used as input data and supervision signals, respectively, and fed into the improved TinyViT student network to perform supervised distillation training. In each training round, the soft label distillation loss is calculated based on the probability distribution distance between the current output of the improved TinyViT student network and the sparse probability label matrix, and backpropagated to update the student network parameters. The similarity between the feature representations generated by the improved TinyViT student network in the intermediate feature layer and the corresponding feature representations in the intermediate layer of the teacher visual Transformer network is evaluated, and a feature preservation loss is constructed and added to the total loss function. During training, the distillation error of each training sample is recorded, and a difficulty guidance factor is constructed based on the historical mean of the distillation error to dynamically weight the distillation loss of each sample. The learning rate and distillation weight factor are adjusted according to the cumulative training rounds and the preset stages. The convergence state is judged based on the loss decrease in adjacent training rounds. Several rounds of parameter updates are performed, and finally a pre-distilled student weight set is generated.

6. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 5, characterized in that, The steps of performing dynamic weighted training on difficult samples include: calculating the distillation error of each training sample based on the difference between the fine-tuned student weight set and the teacher network output; generating a difficulty guidance factor based on the historical mean and volatility of the distillation error; and dynamically weighting the distillation loss of each sample according to the difficulty guidance factor. By weighting the distillation loss, the training effect of the student network on long-tailed samples is optimized.

7. The method for training and optimizing a large-scale model of the thermal coal industry based on knowledge distillation according to claim 6, characterized in that, The steps of performing octet quantization-based perceptual training and channel pruning on the final optimized student weight set include: While maintaining the accuracy of the original model, the final optimized student weight set is subjected to eight-bit integer quantization perceptual training, the dynamic range of each layer weight and activation tensor is recorded, the corresponding quantization scaling factor and zero point value are generated, and a low bit width computation graph is constructed. Channel pruning is performed on the output channels of each convolutional layer to remove low-importance channels and simultaneously adjust the weight dimension and bias structure. After quantization and pruning, a compressed and lightweight large model of the thermal coal industry is exported, deployed to edge computing nodes, and loaded into the inference engine to support real-time inference and output of multimodal data of the thermal coal industry.

Images