Power system large model parameter scale compression and high performance computing resource optimization method

Abstract

The invention relates to the technical field of electric power system intelligence, in particular to an electric power system large model parameter scale compression and high-performance computing resource optimization method, which comprises the steps of obtaining a pre-trained large model, generating a lightweight small model by model distillation, sparsely cutting an optimized structure, performing fine tuning training to improve performance and the like. Through the technical means of model distillation, sparse clipping, semi-supervised fine tuning and the like, the model parameter quantity is compressed to 10%-30% of the original scale, and the reasoning delay is reduced by 40% or above. Meanwhile, the lightweight model shows high applicability and accuracy in scenes such as electric power intelligent questioning and answering, report generation and new energy prediction, and technical support is provided for intelligent transformation of the electric power industry.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of power system and artificial intelligence, and particularly relates to a method for parameter scale compression and high-performance computing resource optimization of a large power system model. BACKGROUND

[0002] In the intelligent transformation of the power system, a large-scale pre-training model can be constructed to effectively mine knowledge in the power industry and provide support for intelligent question answering, report generation, and new energy prediction scenarios. However, the application of large models has high requirements for computing resources, especially in terms of latency and GPU resource occupation during the inference phase. In traditional methods, there is a significant correlation between the allocation of high-performance computing resources and model complexity, and there is still a large optimization space for maintaining performance while reducing the parameter scale of the model.

[0003] Methods based on model compression technology are an emerging research direction that can reduce the number of model parameters and improve inference efficiency to some extent through distillation, sparsification, and pruning. This technology, combined with domain adaptation and a small amount of labeled data fine-tuning, can further enhance the performance of lightweight models in specific scenarios. For example, in the power industry, injecting operation error prevention rule libraries into the semantic understanding process of the model can help improve the term resolution capability. At the same time, the introduction of semi-supervised learning mechanisms enables the model to achieve high accuracy under the condition of a small amount of labeled data.

[0004] Existing model compression and optimization methods still face the following challenges in practical applications:

[0005] Integrity of knowledge transfer in model distillation: Some intermediate layer features may be lost during distillation, resulting in a decline in the performance of lightweight models in complex tasks.

[0006] Precision control of sparse pruning: Excessive pruning may remove key weight channels, affecting the inference accuracy of the model, while insufficient pruning may not significantly reduce resource consumption.

[0007] Adaptability of few-shot learning: In sub-scenarios, the quality and coverage of a small amount of labeled data have an important impact on model fine-tuning, but how to fully exploit the value of unlabeled data still needs further exploration.

[0008] Therefore, the present application is proposed. SUMMARY

[0009] The purpose of the present application is to provide a power system large model parameter scale compression and high-performance computing resource optimization method, which solves the problem of excessive consumption of computing resources and excessive delay of reasoning caused by high model complexity in the prior art through model distillation, sparse pruning and fine-tuning training and other technical means. This method reduces the model parameter scale while improving the performance of the lightweight model in the power industry sub-scene through the full use of a small amount of labeled data and domain adaptation processing.

[0010] To achieve the above purpose, the present application provides a power system large model parameter scale compression and high-performance computing resource optimization method, comprising the following steps:

[0011] Step S1, obtaining a power system pre-training large model, which is generated based on power industry technical standard documents and semantic mining corpus, and injecting operation error prevention rule library knowledge to enhance its understanding ability of power terminology;

[0012] Step S2, using model distillation technology to compress the pre-training large model to generate a lightweight small model;

[0013] Step S3, performing structured sparse pruning and filter pruning on the lightweight small model to remove redundant weight channels and optimize the model structure;

[0014] Step S4, fine-tuning the pruned model using a small amount of labeled data in the power industry sub-scene, and dynamically adjusting the loss function weight combined with unlabeled data to further improve the performance of the model in the specific scene;

[0015] Preferably, the step S1 specifically includes the following contents:

[0016] A corpus set containing power industry technical standard documents, professional term table and operation error prevention rule library is constructed for training the initial large model; After the corpus set is processed by word segmentation, stop word removal and vectorization, a large model with preliminary semantic understanding ability is generated through a deep learning framework; On this basis, the constraint conditions in the operation error prevention rule library are embedded as additional input into the model, so that the model can parse the terms and logical relationships specific to the power industry; For example, a rule defines that "the closing priority of the circuit breaker is higher than that of the disconnector", and the rule is coded as a feature vector and embedded into the hidden layer of the model, thereby enhancing the model's understanding ability of the power system operation procedures;

[0017] Preferably, the step S2 specifically includes the following contents:

[0018] Model distillation techniques are chosen as compression methods, including at least one of the TinyBERT knowledge distillation framework, DistillBERT inter-layer knowledge transfer, and PKD-BERT intermediate layer feature matching. Taking TinyBERT as an example, the output distribution of the original large model is first used as the knowledge source of the teacher model, and the learning process of the student model is guided by the cross-entropy loss function. Specifically, the soft-label output of the teacher model is mapped to the hidden layer of the student model, enabling the student model to inherit the core knowledge of the teacher model with a smaller number of parameters. In addition, an intermediate layer feature matching mechanism is introduced during the distillation process to ensure that the activation values ​​of the student model in each layer are consistent with those of the teacher model, thereby avoiding the loss of key information.

[0019] Preferably, step S3 specifically includes the following:

[0020] In a lightweight small model, weight channels below a preset threshold are identified, and their corresponding filter components are removed to generate a sparse model structure. Specifically, a threshold τ is set. For each layer weight matrix W of the model, if the L2 norm of a certain channel is less than τ, the channel is considered to contribute little to the overall model and can be pruned. The pruned model structure maintains computational efficiency while reducing GPU memory usage by rearranging the remaining weights. In addition, for the convolutional layers of the model, filter pruning techniques are used to remove redundant filters, further reducing the space complexity of the model.

[0021] Preferably, step S4 specifically includes the following:

[0022] The fine-tuning training phase employs a semi-supervised learning mechanism, mixing unlabeled data from the power industry with a small amount of labeled data as input. Reinforcement learning is used to dynamically adjust the weights of the loss function. Specifically, if the unlabeled dataset is U and the labeled dataset is L, then the total loss function Loss is expressed as:

[0023] Loss = α·Loss L +(1-α)·Loss U

[0024] Wherein, LossL is the supervised loss for labeled data, LossU is the pseudo-label loss for unlabeled data, and α is the dynamic weight coefficient; the value of α is updated in real time through reinforcement learning algorithm, so that it is automatically adjusted in different training stages to balance the contribution ratio of labeled data and unlabeled data; in addition, fewer than 1,000 labeled samples of power scene are used for few-shot learning, and the model accuracy is gradually improved to more than 90% through iterative training.

[0025] Preferably, step S4 further includes the following:

[0026] After the fine-tuning training is completed, the light-weight small model is plug-in deployed and optimized; specifically, after the model is integrated into the terminal device, the service arrangement is performed through the unified gateway module, and the GPU resource occupation rate is monitored in real time; the terminal device supports dynamically loading the following function modules: a business access module, an online test module and a security permission verification module; for example, in the new energy consumption prediction decision-making scene, the business access module is responsible for receiving real-time power grid data, the online test module is used for evaluating the model performance, and the security permission verification module ensures the security of data transmission.

[0027] After the above technical solutions are adopted, the present application has the following beneficial effects compared with the prior art, of course, any product implementing the present application does not necessarily need to achieve all the advantages described below:

[0028] The present application adopts a power system large model parameter scale compression and high-performance computing resource optimization method, effectively reduces the model parameter quantity to 10%-30% of the original large model through model distillation, sparse pruning and fine-tuning training and other technical means, and reduces the inference delay by more than 40%. At the same time, the light-weight model shows high applicability and accuracy in the power system intelligent question and answer service, power professional report automatic generation and new energy consumption prediction decision-making scenes, and provides technical support for the intelligent transformation of the power industry.

[0029] The specific embodiments of the present application will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0030] The drawings in the following description are only some embodiments, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0031] Figure 1 The present application is a power system large model parameter scale compression and high-performance computing resource optimization method flow chart.

[0032] It should be noted that these drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] The present application will now be described in further detail with reference to the accompanying drawings.

[0034] Please refer to Figure 1 In the present embodiment, a power system large model parameter scale compression and high-performance computing resource optimization method is provided, comprising the following steps:

[0035] First, from the construction of the pre-training large model, the model is based on the power industry technical standard documents, professional term table and operation anti-misoperation rule library to generate corpus set, and the preliminary training is completed through the deep learning framework; in actual operation, the construction process of corpus set needs to go through steps such as word segmentation, stop word removal and vectorization processing, so as to ensure that the quality of corpus data meets the model training requirements; for example, in the word segmentation stage, the power industry dedicated word segmentation tool is used to cut the document, and after removing the meaningless stop words, the text is converted into a numerical vector representation form; then, the vectorized corpus is trained through a deep learning framework such as PyTorch or TensorFlow, to generate a large model with preliminary semantic understanding ability; on this basis, the constraint conditions in the operation anti-misoperation rule library are coded as feature vectors and injected into the hidden layer of the model to enhance its understanding ability of power terminology and operation logic; for example, for the rule "the closing priority of circuit breaker is higher than that of disconnector", it is parsed as a set of Boolean logic relations and embedded into the specific hidden layer nodes of the model, so as to realize the modeling of the knowledge specific to the power industry;

[0036] Next, the implementation stage of the model distillation module is entered, which uses TinyBERT knowledge distillation framework, DistillBERT inter-layer knowledge transfer or PKD-BERT intermediate layer feature matching technology to compress the pre-training large model; taking TinyBERT as an example, the output distribution of the teacher model is used as a knowledge source to guide the learning process of the student model; specifically, the soft label output of the teacher model is mapped to the hidden layer of the student model through the cross-entropy loss function, so that the student model can inherit the core knowledge of the teacher model under the condition of smaller parameter quantity; in order to ensure that the key information is not lost in the distillation process, an intermediate layer feature matching mechanism is introduced; this mechanism constrains the consistency of the activation values of each layer of the teacher model and the student model, ensuring that the student model can accurately reproduce the behavior characteristics of the teacher model; for example, in a certain layer, if the mean value of the activation value of the teacher model is 0.8 and the variance is 0.1, then through regularization means, the activation value of the student model is made as close to this distribution range as possible, so as to ensure the stability of the model performance;

[0037] The sparse pruning module is responsible for implementing structured sparse pruning and filter pruning on the lightweight small model, removing redundant weight channels and optimizing the model structure. In actual operation, a threshold τ is set to judge the importance of the weight channel. For each layer weight matrix W of the model, the L2 norm of each channel is calculated and compared with τ. If the L2 norm of a channel is less than τ, it is considered that the channel contributes less to the overall model and can be pruned. After pruning, the remaining weights need to be rearranged to maintain computational efficiency and reduce GPU memory usage. For the convolutional layer part, filter pruning technology is used to further reduce the spatial complexity of the model. For example, in a certain convolutional layer, the original number of filters is 64, and after pruning, only 48 filters are retained, thereby significantly reducing the parameter quantity and computational overhead of the model.

[0038] The fine-tuning training module fine-tunes the pruned model using a small amount of labeled data in the power industry sub-scene, and dynamically adjusts the loss function weight combined with unlabeled data. In actual operation, let U be the unlabeled data set and L be the labeled data set. The total loss function Loss is defined as α times the supervised loss LossL of the labeled data plus 1 minus α times the pseudo-label loss LossU of the unlabeled data. Where α is the dynamic weight coefficient, which is updated in real time to balance the contribution proportion of labeled data and unlabeled data through reinforcement learning algorithm. For example, in the new energy consumption prediction decision-making scene, less than 1000 labeled samples are used for few-shot learning, and the model accuracy is gradually improved to more than 90% through iterative training. In addition, the fine-tuning training process also combines a semi-supervised learning mechanism to fully utilize unlabeled data to expand the generalization ability of the model.

[0039] Finally, the implementation phase of the plug-in deployment module is entered, which is responsible for integrating the lightweight small model into the terminal device and performing service orchestration. In actual operation, the model is deployed through a unified gateway module, and the terminal device supports dynamic loading of business access modules, online testing modules, and security permission verification modules and other functional components. For example, in the new energy consumption prediction decision-making scene, the business access module is responsible for receiving real-time power grid data, the online testing module is used to evaluate the model performance, and the security permission verification module ensures the security of data transmission. In addition, the deployment process monitors the GPU resource occupancy rate in real time, and dynamically adjusts the running parameters of the model according to the hardware configuration of the terminal device to ensure the stable operation of the model in different hardware environments.

[0040] The various modules in the above implementation process cooperate closely through data flow and control flow to achieve the goal of power system large model parameter scale compression and high-performance computing resource optimization; for example, the output of the pre-trained large model is used as the input of the model distillation module, the distilled lightweight small model is passed to the sparse pruning module for further optimization, the pruned model is passed to the fine-tuning training module for scene adaptation, and finally the landing application of the model is completed through the plug-in deployment module; the specific operation of each module must be strictly performed according to the above process to ensure the integrity and effectiveness of the entire technical solution;

[0041] In order to better enable relevant persons in the technical field to fully understand and implement the present application, the specific implementation principles of the present application are supplemented in the following in conjunction with a specific application scenario;

[0042] In the power system intelligent question and answer service scenario, a basic model with preliminary semantic understanding ability is first generated through a pre-trained large model; the construction of the model is based on a corpus set generated from technical standard documents, professional term tables and operation anti-misoperation rule libraries in the power industry, and data preparation is completed through steps such as word segmentation, stop word removal and vectorization processing; for example, for the rule "the closing priority of the circuit breaker is higher than that of the disconnector", it is parsed into a Boolean logical relationship and embedded into a specific node in the hidden layer of the model to enhance the model's understanding ability of power terminology and operation logic; on this basis, the model distillation module uses the TinyBERT knowledge distillation framework to compress the pre-trained large model; the soft label output of the teacher model is mapped to the hidden layer of the student model through the cross-entropy loss function, and an intermediate layer feature matching mechanism is introduced to ensure that the activation value distribution of the student model at each layer is consistent with that of the teacher model; for example, in a certain layer, if the mean of the activation value of the teacher model is 0.8 and the variance is 0.1, then the activation value of the student model is made as close to this range as possible through regularization, thereby ensuring the performance stability of the lightweight small model in complex tasks;

[0043] Subsequently, the implementation phase of the sparse pruning module is entered, the weight matrix of the lightweight small model is analyzed, and a threshold τ is set to judge the importance of the weight channel; for each layer weight matrix W of the model, the L2 norm of each channel is calculated and compared with τ; if the L2 norm of a certain channel is less than τ, it is considered that the channel contributes less to the overall model and can be pruned; after pruning, the remaining weights need to be rearranged to maintain computational efficiency and reduce GPU memory usage; for example, in a certain convolutional layer, the original number of filters is 64, and after pruning, only 48 filters are retained, thereby significantly reducing the spatial complexity and computational overhead of the model;

[0044] Next, the fine-tuning training module is implemented, and the pruned model is fine-tuned using a small amount of labeled data in the power industry sub-scene; Let U be the unlabeled data set, and L be the labeled data set. The total loss function Loss is defined as alpha times the supervised loss LossL of the labeled data plus 1 minus alpha times the pseudo-label loss LossU of the unlabeled data; Where alpha is a dynamic weight coefficient, which is updated in real time by a reinforcement learning algorithm to balance the contribution ratio of labeled data and unlabeled data; For example, in the intelligent question answering scene, less than 1000 labeled samples are used for few-shot learning, and the model accuracy is gradually improved to more than 90% through iterative training; In addition, the semi-supervised learning mechanism is combined in the fine-tuning training process to fully utilize the unlabeled data to expand the generalization ability of the model;

[0045] Finally, the plug-in deployment module is implemented, and the lightweight small model is integrated into the terminal device and arranged for service; In actual operation, the model is deployed through a unified gateway module, and the terminal device supports dynamic loading of business access modules, online test modules, and security permission verification modules and other functional components; For example, in the intelligent question answering service scene, the business access module is responsible for receiving user questions, the online test module is used to evaluate the model performance, and the security permission verification module ensures the security of data transmission; In addition, the GPU resource occupancy rate is monitored in real time during deployment, and the running parameters of the model are dynamically adjusted according to the hardware configuration of the terminal device to ensure the stable operation of the model in different hardware environments;

[0046] The various modules in the above implementation process cooperate closely through data flow and control flow to achieve the goal of parameter scale compression and high-performance computing resource optimization of the power system large model; For example, the output of the pre-trained large model is used as the input of the model distillation module, the distilled lightweight small model is passed to the sparse pruning module for further optimization, the pruned model is passed to the fine-tuning training module for scene adaptation, and finally the plug-in deployment module is used to complete the landing application of the model; The specific operation of each module must be strictly performed according to the above process to ensure the integrity and effectiveness of the entire technical solution.

[0047] The contents not described in detail in the specification are all existing technologies known to those skilled in the art, and the model parameters of each electric appliance are not specifically limited, and conventional equipment can be used; In the present technical solution, the electric appliance control elements not mentioned belong to existing technologies, and therefore are not shown in the drawings, and will not be described here.

[0048] The present application is not limited to the above embodiments, and anyone should know that any structural changes made under the inspiration of the present application fall within the protection scope of the present application. The technical, shape and structure parts not described in detail in the present application are all known technologies.

Claims

1. A power system large model parameter scale compression and high-performance computing resource optimization method, characterized in that, The method comprises the following steps: Step S1, obtaining a pre-trained large model, which is generated based on power industry technical standard documents, professional term tables and operation anti-misoperation rule libraries, and is preliminarily trained through a deep learning framework; Step S2, compressing the pre-trained large model by using a model distillation module to generate a light-weight small model; Step S3, implementing structural sparse pruning and filter pruning on the light-weight small model by using a sparse pruning module to remove redundant weight channels and optimize the model structure; Step S4, fine-tuning the pruned model by using a fine-tuning training module, and dynamically adjusting the loss function weight in combination with unannotated data; Step S5, integrating the fine-tuned light-weight small model into a terminal device by using a plug-in deployment module and performing service orchestration.

2. The method of claim 1, wherein, The step S1 specifically comprises: A corpus set containing power industry technical standard documents, professional term tables and operation anti-misoperation rule libraries is constructed, and after word segmentation, stop word removal and vectorization processing are performed on the corpus set, a large model with preliminary semantic understanding ability is generated through a deep learning framework; The constraint conditions in the operation anti-misoperation rule library are encoded into feature vectors and injected into the hidden layer of the model.

3. The method of claim 1, wherein, The step S2 specifically comprises: One or more of the TinyBERT knowledge distillation framework, the DistillBERT inter-layer knowledge migration or the PKD-BERT middle-layer feature matching technology is selected to compress the pre-trained large model; The learning process of the student model is guided by using a cross-entropy loss function, and a middle-layer feature matching mechanism is introduced to ensure that the activation values of each layer of the student model are consistent with those of the teacher model.

4. The method of claim 1, wherein, The step S3 specifically comprises: A threshold τ is set, and for each weight matrix W of the model, if the L2 norm of a channel is less than τ, the filter component corresponding to the channel is removed; The remaining weights of the pruned model structure are rearranged to maintain computational efficiency, and the filter pruning technology is used to remove redundant filters for the convolutional layer.

5. The method of claim 1, wherein, The step S4 specifically comprises: Let the unannotated data set be U, the annotated data set be L, and the total loss function Loss be defined as α times the supervised loss LossL of the annotated data plus (1-α) times the pseudo-label loss LossU of the unannotated data; The value of α is updated in real time by using a reinforcement learning algorithm to balance the contribution proportion of the annotated data and the unannotated data.

6. The method of claim 5, wherein, The step S4 further comprises: Few-shot learning is performed using less than 1000 annotated samples, and the model accuracy is gradually improved through iterative training.

7. The method of claim 1, wherein, The step S5 specifically comprises: After the light-weight small model is integrated into the terminal device, service orchestration is performed through a unified gateway module; The terminal device supports dynamically loading a business access module, an online test module and a security permission verification module.

8. The method of claim 7, wherein, The business access module is responsible for receiving real-time power grid data, the online test module is used to evaluate the model performance, and the security permission verification module ensures the security of data transmission.

9. The method of claim 1, wherein, In the step S2, the middle-layer feature matching mechanism performs consistency constraint on the activation values of each layer of the teacher model and the student model, so as to ensure that the student model can accurately reproduce the behavior characteristics of the teacher model.

10. The method of claim 1, wherein, In the step S3, the cropped model structure is optimized in terms of computing resources by reducing GPU memory occupation.

Images