Electric power system specific subdivision scene model construction method based on model distillation

Abstract

The invention relates to the technical field of power system intellectualization, in particular to a power system specific subdivision scene model construction method based on model distillation. The method comprises the steps of obtaining a pre-trained large model in the electric power field, migrating knowledge to a lightweight student model through a model distillation technology, applying structured sparse clipping and filter clipping compression parameter scale, and using a small amount of annotated data to finely adapt to a specific scene. According to the method, the model complexity and the calculation cost can be remarkably reduced, meanwhile, the high-precision reasoning performance is ensured, a customized lightweight solution is provided for a power business scene in a resource limited environment, and intelligent development of a power system is assisted.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of power system intelligence, and in particular relates to a power system specific sub-scene model construction method based on model distillation. BACKGROUND

[0002] With the development of intelligent power systems, natural language processing technology based on artificial intelligence has been widely applied in power business scenarios. For example, semantic understanding of power dispatching instructions, device operation error prevention verification, new energy consumption report generation, and relay protection setting value specification analysis tasks all require efficient and accurate text analysis and generation capabilities. However, existing general natural language processing models are usually large in size and require high computational resources, making it difficult to meet the real-time and lightweight requirements of power systems in actual deployment. At the same time, the power industry has a high degree of professionalism, and its text data contains a large number of industry terms and specific rules, which puts higher requirements on the domain adaptability of the model.

[0003] Currently, there are two technical paths for power field applications: one is to directly use general pre-trained large models for fine-tuning, but this method has the problem of excessive model parameter size and low inference efficiency, especially in resource-constrained edge computing environments. The second is to develop specialized small models, but such models often suffer from performance degradation due to insufficient training data. In addition, existing technologies lack effective knowledge transfer mechanisms, resulting in low efficiency of knowledge transfer from large-scale pre-trained models to lightweight models, and the complexity of domain knowledge cannot be fully retained.

[0004] On the other hand, in power sub-scenarios, the cost of obtaining labeled data is high and the amount of data is limited, which further limits the application effect of traditional deep learning methods. Although few-shot learning and zero-shot learning mechanisms can alleviate the problem of data insufficiency to some extent, how to ensure model performance while achieving lightweight and efficient inference remains a technical problem to be solved.

[0005] In summary, the existing technology in the application of power system specific sub-scenarios faces the following challenges: model size is too large, leading to deployment difficulties, low efficiency of domain knowledge transfer, insufficient small sample learning ability, and limited support for professional terms and rules. Therefore, developing a lightweight model construction method that combines model distillation, structured sparse pruning, filter pruning, and other technologies to meet the needs of power system specific business scenarios has important practical significance and technical value. SUMMARY

[0006] The technical problems to be solved by the present application are to overcome the deficiencies of the prior art and provide a power system specific subdivision scene model construction method based on model distillation, which solves the problems raised in the background art.

[0007] To solve the above technical problems, the basic idea of the technical solution adopted by the present application is:

[0008] A power system specific subdivision scene model construction method based on model distillation, comprising the following steps:

[0009] Step S1, obtain a pre-training large model in the power field, which is trained based on massive power knowledge data and has the ability to understand and generate power text; the pre-training large model adopts a Transformer architecture and introduces a semantic constraint mechanism specific to the power field during training to enhance its modeling ability for professional terms and contextual relationships;

[0010] Step S2, use model distillation technology to transfer the knowledge of the power field pre-training large model to a lightweight student model, wherein at least one of TinyBERT, DistillBERT or PKD-BERT can be selected as the distillation framework; a dynamic temperature coefficient adjustment strategy is used in the distillation process to optimize the soft label transfer efficiency, and the hidden layer output distribution of the teacher model and the student model is aligned through a knowledge distillation loss function, which is composed of KL divergence distance and mean square error loss, and the specific formula is as follows:

[0011] L distill =α·KL(T(y teacher ),T(y student ))+(1-α)·MSE(H teacher ,H student )

[0012] Wherein, T(□) represents a temperature scaling function, y teacher and y student are the prediction probability distributions of the teacher model and the student model respectively, H teacher and H student are the hidden layer outputs of the two, and a is a weight coefficient used to balance the two losses;

[0013] Step S3, apply structured sparse pruning and filter pruning to the distilled student model to further compress the model parameter size; structured sparse pruning identifies neuron connections with a contribution lower than a preset threshold, performs block sparse processing on the weight matrix of the multi-head attention layer, and removes redundant parameter blocks; filter pruning analyzes the channel importance scores of the convolution layer, removes feature channels and their associated filters with scores lower than the pruning threshold, and the pruning ratio is dynamically adjusted by the model performance evaluation index;

[0014] Step S4, using a small amount of labeled data of power subdivision scene to fine-tune the compressed model, outputting a lightweight inference model for a specific power business scene; the fine-tuning adopts a few-shot learning mechanism, and the amount of training data used is not more than 5% of the amount of data in the pre-training stage, and the distilled compressed model can be directly used to perform the power text analysis task in the zero-shot scene without additional fine-tuning.

[0015] Preferably, in the step S1, the training data of the power field pre-training large model includes power dispatch instruction text and semantic annotation, device operation anti-misoperation rule description, new energy output prediction report segment, and key items of power technology standard documents, so as to ensure that the model can comprehensively cover the diversified application scenarios in the power field.

[0016] Preferably, in the step S3, the specific implementation of the structured sparse pruning is that: first, the importance scores of each weight block of the multi-head attention layer are calculated, and the importance score is defined as:

[0017]

[0018] wherein, W block is the matrix of the current weight block, ||□|| F is the Frobenius norm, and N is the total number of weight blocks; then, according to the score ranking, the weight block with the lowest score is removed to realize the sparsification.

[0019] Preferably, in the step S4, the data used for fine-tuning covers specific power business scenarios such as power dispatch instruction semantic understanding, operation ticket anti-misoperation verification, new energy consumption report generation, and relay protection setting specification analysis, so as to ensure the adaptability and generalization ability of the model in actual application.

[0020] Preferably, when the lightweight inference model is deployed, the power business request is received through a unified gateway module, the lightweight inference model is dynamically called by using a service arrangement module, and the industry term library support is extended through a plug-in function module, so as to realize efficient and flexible power business inference service.

[0021] After the above technical solution is adopted, the present application has the following beneficial effects compared with the prior art. Of course, any product implementing the present application does not necessarily need to achieve all the advantages described below:

[0022] The application adopts the above-mentioned power system specific subdivision scene model construction method based on model distillation, significantly reduces the model complexity and calculation cost through efficient model distillation, structured sparse pruning and filter pruning and other technical means, and realizes precise adaptation to specific power business scenes by combining a small amount of labeled data fine-tuning, exhibits excellent inference performance and deployment flexibility in a resource-limited environment, and provides strong technical support for the intelligent development of the power system.

[0023] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings in the following description are only some embodiments, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0025] Figure 1 A flowchart of a specific subdivision scene model construction method for a power system based on model distillation.

[0026] It should be noted that these drawings and written descriptions are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0027] The application will now be further described in detail with reference to the accompanying drawings;

[0028] Please refer to Figure 1 As shown in the drawings, a specific subdivision scene model construction method for a power system based on model distillation is provided in this embodiment, the core of which is to efficiently transfer the knowledge of a pre-trained large model to a lightweight child model through model distillation technology, and to significantly reduce the model complexity and calculation cost by combining structured sparse pruning, filter pruning and a small amount of labeled data fine-tuning and other technical means, while ensuring high-precision inference performance in a resource-limited environment; the following will be described in detail in combination with the accompanying Figure 1 The specific embodiments of the application will be described in detail.

[0029] According to Figure 1The overall process shown, the implementation process of the present application starts from obtaining a pre-training large model in the field of power, that is, step S1; in this step, first, a large amount of power knowledge data needs to be prepared, which includes but is not limited to power dispatch instruction text and semantic annotation, equipment operation error prevention rule description, new energy output prediction report fragment, and key entries of power technology standard documents; the selection of these data ensures that the model can comprehensively cover the diversified application scenarios in the power field, thereby enhancing its modeling capability for professional terms and context relationships; specifically, the pre-training large model adopts the Transformer architecture, and introduces a semantic constraint mechanism specific to the power field during training; for example, in the training stage, the key terms in the power dispatch instructions are weighted to make the model better capture the semantic information of these terms; in addition, in order to improve the model's understanding ability of long text, a dynamic position encoding mechanism is also introduced to adapt to input sequences of different lengths;

[0030] After completing the construction of the pre-training large model, step S2 is entered, that is, the knowledge of the teacher model is transferred to the lightweight student model through model distillation technology; in this step, the distillation framework can select at least one of TinyBERT, DistillBERT or PKD-BERT; the core of distillation is to make the student model as close as possible to the output distribution of the teacher model through soft label transmission; to optimize the efficiency of soft label transmission, the present application adopts a dynamic temperature coefficient adjustment strategy; specifically, the temperature coefficient T is dynamically adjusted according to the loss value of the current training stage during the distillation process, and the initial value is set to a higher temperature to smooth the probability distribution, and the temperature is gradually reduced to highlight the importance of the true label as the training progresses; the knowledge distillation loss function used in the distillation process is composed of KL divergence distance and mean square error loss, and the formula is as follows:

[0031] L distill =α·KL(T(y teacher ),T(y student ))+(1-α)·MSE(H teacher ,H student )

[0032] Where T(□) represents the temperature scaling function, y teacher and y student are the predicted probability distributions of the teacher model and the student model respectively, H teacher and H student are the hidden layer outputs of the two, and a is the weight coefficient for balancing the two losses; through the design of the above loss function, the hidden layer output distribution of the teacher model and the student model can be effectively aligned, thereby realizing effective knowledge transfer;

[0033] After completing model distillation, step S3 is entered, that is, structured sparse pruning and filter pruning are applied to the distilled student model to further compress the model parameter size; the goal of structured sparse pruning is to identify and remove low-contribution neuron connections, thereby achieving block sparse processing of the model; the specific implementation is as follows: first, the importance scores of each weight block of the multi-head attention layer are calculated, and the importance score is defined as:

[0034]

[0035] where W block is the matrix of the current weight block, ||□|| F is the Frobenius norm, and N is the total number of weight blocks. Subsequently, according to the score ranking, the weight block with the lowest score is removed to achieve sparse processing; in practical applications, the sparse ratio is usually set to 10% to 30%, and the specific value needs to be dynamically adjusted according to the model performance evaluation index; for filter pruning, the goal is to analyze the channel importance scores of the convolution layer and remove feature channels and their associated filters whose scores are lower than the pruning threshold; the pruning threshold is set based on the importance score distribution of the channels, and channels with scores in the last 5% to 10% are usually removed; through the above two pruning methods, the model parameter size can be significantly reduced while the core performance of the model is preserved as much as possible.

[0036] After completing structured sparse pruning and filter pruning, step S4 is entered, that is, the compressed model is fine-tuned using a small amount of labeled data of power subdivision scenarios to output a lightweight inference model for specific power business scenarios; the fine-tuning uses a few-shot learning mechanism, and the amount of training data used is no more than 5% of the data amount in the pre-training phase; for example, for the power dispatch instruction semantic understanding task, a small amount of samples containing typical dispatch instructions and their semantic annotations are selected for fine-tuning; for the operation ticket anti-misoperation verification task, operation ticket data covering common anti-misoperation rules are selected for training; in addition, the fine-tuning supports directly using the distilled and compressed model to perform power text parsing tasks in a zero-shot scenario without additional fine-tuning; this design enables the model to maintain a certain generalization ability in the case of data scarcity, meeting the actual application requirements.

[0037] In the implementation process, the deployment of the lightweight inference model is also carefully designed; when the model is deployed, the power business request is received through the unified gateway module, the lightweight inference model is dynamically called by using the service arrangement module, and the industry term library support is expanded through the plug-in function module, so that efficient and flexible power business inference service is realized; for example, when processing the power dispatching instruction, the unified gateway module first performs preliminary analysis on the received instruction, and transmits the analysis result to the service arrangement module; the service arrangement module dynamically calls the corresponding lightweight inference model according to the instruction type, and combines the industry term library provided by the plug-in function module to perform deep analysis on the instruction, and finally outputs the result meeting the business demand.

[0038] To sum up, by means of the technical means such as efficient model distillation, structured sparse pruning and filter pruning, the model complexity and calculation cost are significantly reduced, and the precise adaptation to specific power business scenes is realized by combining a small amount of labeled data fine-tuning; in the resource-limited environment, the present application shows excellent inference performance and deployment flexibility, and provides strong technical support for the intelligent development of the power system.

[0039] The present application is not limited to the above-mentioned embodiments, and anyone should know that the structural changes made under the inspiration of the present application fall within the protection scope of the present application, and any technical solutions with the same or similar technical solutions as the present application fall within the protection scope of the present application; the technical, shape and structure parts not described in detail in the present application are all known technologies.

Claims

1. A method for constructing a model distillation-based power system specific fine-grained scenario model, characterized in that, The method comprises the following steps: Step S1, obtaining a power field pre-training large model, which is trained based on massive power knowledge data, adopts a Transformer architecture, and introduces a semantic constraint mechanism specific to the power field during training; Step S2, migrating the knowledge of the power field pre-training large model to a lightweight student model through a model distillation technique, selecting at least one of TinyBERT, DistillBERT or PKD-BERT as the distillation framework, optimizing the soft label transfer efficiency using a dynamic temperature coefficient adjustment strategy during the distillation process, and aligning the hidden layer output distribution of the teacher model and the student model through a loss function; Step S3, applying structured sparse pruning and filter pruning to the distilled student model to further compress the model parameter size; Step S4, fine-tuning the compressed model using a small amount of labeled data of power sub-scenarios to output a lightweight inference model for specific power business scenarios.

2. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, In step S1, the training data of the power field pre-training large model includes power dispatch instruction text and semantic annotations, device operation error prevention rule descriptions, new energy output prediction report segments, and key entries of power technology standard documents.

3. The model distillation-based power system specific fine-grained scenario model construction method of claim 1, wherein, In step S2, the loss function is composed of KL divergence distance and mean square error loss, and the specific formula is: L distill = a - KL(T(y teacher ), T(y student )) + (1 - a) - MSE(H teacher , H student ) where, denotes a temperature scaling function, y teacher and y student are the predictive probability distributions of the teacher and student models, respectively, H teacher and H student are the hidden layer outputs of both, and a is a weighting coefficient to balance the two losses.

4. The model distillation-based power system specific fine-grained scenario model construction method of claim 1, wherein, In step S3, the specific implementation of structured sparse pruning is as follows: first, calculate the importance score of each weight block of the multi-head attention layer, and define the importance score as: where W block is the matrix of the current weight block, ||□| F is the Frobenius norm, and N is the total number of weight blocks; then the weight blocks with the lowest scores are removed to achieve sparsification according to the score ranking.

5. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, In step S3, the specific implementation of filter pruning is as follows: analyze the channel importance score of the convolution layer, remove the feature channels and their associated filters whose scores are lower than the pruning threshold, and dynamically adjust the pruning ratio according to the model performance evaluation index.

6. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, In step S4, the fine-tuning uses a few-shot learning mechanism, and the amount of training data used is not more than 5% of the amount of data in the pre-training phase.

7. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, In step S4, the fine-tuning uses data covering power dispatch instruction semantic understanding, operation ticket error prevention verification, new energy consumption report generation, and relay protection setting specification analysis for specific power business scenarios.

8. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, When deploying the lightweight inference model, a unified gateway module receives power business requests, a service orchestration module dynamically calls the lightweight inference model, and a plug-in function module extends an industry term library for support.

9. The model distillation-based power system specific fine-grained scenario model construction method according to claim 1, characterized in that, In step S2, the specific implementation of the dynamic temperature coefficient adjustment strategy is as follows: the initial temperature is set to a high value to smooth the probability distribution, and the temperature is gradually reduced as the training progresses to highlight the importance of the true label.

10. The model distillation-based power system specific fine-grained scenario model construction method of claim 1, wherein, In step S4, the distilled and compressed model can be directly used to perform power text analysis tasks in a zero-shot scenario without additional fine-tuning.

Images