Method for optimizing neural network model based on knowledge distillation technology

Abstract

The invention discloses a method for optimizing a neural network model based on a knowledge distillation technology, which is characterized in that a plurality of teacher networks are established to respectively learn different physical laws, and physical constraints are embedded in each network, so that the learning ability and performance of student networks in complex power dispatching problems are remarkably improved. Besides, only the teacher network of which the task performance is superior to that of the student network is selected for knowledge transmission, so that the interference of low-efficiency and useless information is avoided, and the learning efficiency and the result reliability are improved. Meanwhile, the physical law serves as guidance information to penetrate through the knowledge transmission process, it is ensured that the student network is not limited to a learning data mode and can follow physical constraints in the power system, and therefore the interpretability and stability of the model are enhanced.

Description

Technical Field

[0001] The invention belongs to the field of power system dispatching, and in particular, relates to a method for optimizing a neural network model based on knowledge distillation technology. Background Art

[0002] In the field of power system dispatch, deep neural networks have demonstrated excellent learning capabilities when dealing with complex dispatch problems, especially in data-driven prediction and optimization tasks. However, power systems involve many physical laws, such as power conservation, voltage stability, and flow constraints, which are often difficult to fully express in data patterns. Since deep neural networks rely on data patterns during training and lack clear guidance from physical constraints, it is difficult for the model to effectively capture the physical characteristics of these systems in actual dispatch. Therefore, when dealing with tasks such as load fluctuations, fault recovery, and emergency dispatch that may occur in actual power systems, the model's reasoning ability and stability are often limited.

[0003] Although traditional supervised learning methods can use a large amount of labeled data to train models, this data is not enough to fully express the complex physical laws in the power system. On this basis, the model can only learn the surface data features and it is difficult to grasp the underlying physical essence. This learning method that relies on pure data drive also affects the generalization ability of the model, especially when dealing with actual dispatching scenarios that are not completely consistent with the distribution of training data, the model shows instability.

[0004] Although existing knowledge distillation techniques transfer knowledge from the teacher network to the student network, they usually only focus on the transfer of data patterns, while ignoring the guidance and constraints of the physical laws that are crucial to the power system. As a result, the student network may deviate from physical reality in complex power dispatching tasks, making it difficult for its decisions to ensure the stability and reliability of the system in practical applications.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a method for optimizing a neural network model based on knowledge distillation technology, thereby solving the problems raised in the above-mentioned background technology.

[0007] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0008] A method for optimizing a neural network model based on knowledge distillation technology comprises the following steps:

[0009] Step S1: First, multiple teacher networks are established, each focusing on a different scheduling task, and then trained independently and embedded with power balance and voltage stability physical constraints;

[0010] Step S2: Evaluate the performance of all teacher networks and select the teacher networks that perform better than the student networks on specific scheduling tasks to ensure that knowledge transfer is only carried out from the teacher networks with better performance;

[0011] Step S3: During the knowledge transfer process, the output of the teacher network is temperature-conditioned and then passed to the student network, ensuring that the student network not only learns the data pattern but also follows the physical laws of the power system;

[0012] Step S4: During the knowledge distillation process, the hyperparameters including the temperature parameter, the hard loss and the soft loss ratio are adjusted, and the strategy of gradually reducing the soft loss ratio is adopted. During the training process of the student network, the supervision signal of the teacher network is gradually projected into the parameter space of the student network in combination with the progressive knowledge distillation method, and the training objective is decomposed into multiple intermediate objectives, which gradually guides the parameter adjustment and optimization of the student network.

[0013] Optionally, multiple teacher network models are established and trained to learn different physical laws respectively, and each teacher network is trained independently and embedded with physical constraints. Then, the steps of establishing a student network model to be optimized are:

[0014] Step S101: Select a convolutional neural network as the neural network architecture, then set initial weights, bias values, and learning rates for each teacher network, and then independently train multiple teacher networks to optimize their loss functions;

[0015] Step S102: In the training of the teacher network, a loss term related to physical laws is introduced on the basis of the original loss function and the physical law loss is combined with the conventional loss to form a composite loss function, which is expressed as: L total =L task +λ∑(P pred -P true ) 2 , where L task is the task loss, L phys is the physical law loss, λ is the coefficient that controls the weight of the two, P pred is the physical quantity predicted by the model, P true It is a real physical quantity, ensuring the conservation relationship in the output;

[0016] Step S103: By formula The mathematical expressions of power conservation and voltage stability are embedded in the network’s loss function. The composite loss function containing physical constraints is used to independently train the teacher network so that it can follow the physical laws of the power system while learning the data pattern. j | is the voltage at node j, V min and Vmax are the lower and upper limits of the voltage, and γ is the weighting factor;

[0017] Step S104: During and after the training process, the model output is verified to ensure that it meets the physical constraints of power balance and voltage stability.

[0018] Optionally, a performance-aware mutual knowledge distillation method is used to evaluate the performance of multiple teacher networks and only select the teacher network with better performance than the student network to transfer knowledge in the following steps:

[0019] Step S201: Evaluate the performance of all teacher networks and student networks on the same data set, and use predefined performance indicators to calculate the performance of each network on the task. At this time, evaluate the gap between the initial performance of the student network and the output of multiple teacher networks. Then, based on the results of the performance evaluation, select the teacher network with better performance than the student network.

[0020] Step S202: Among the selected high-quality teacher networks, the knowledge of the teacher network is transferred to the student network through knowledge distillation. The softmax output of the teacher network is used as the learning target of the student network. There are multiple excellent teacher networks. The outputs of these teacher networks are integrated through weighted fusion. The expression is: Among them, P fused is the softmax output after weighted fusion of multiple teacher networks, N is the number of teacher networks, represents the softmax output of the i-th teacher network, w i is the weight of the i-th teacher network, satisfying

[0021] Step S203: During the training process, the performance of the student network is regularly re-evaluated and compared with the teacher network. As the performance of the student network improves, some teacher networks no longer perform better than the student network. The selection of teacher networks is dynamically updated and teacher networks that are no longer better than the student network are eliminated.

[0022] Step S204: As the performance of the student network gradually surpasses that of the teacher network, the weight of the soft loss is lowered, the hard loss ratio of the student network to the true label is increased, and the independent learning ability of the student network is gradually enhanced, so that it can eventually perform tasks without relying on the teacher network.

[0023] Optionally, in step S2, during the data transfer process, multiple key anchor points in the teacher network training path are selected and the output is processed in blocks at the anchor points to transfer local and global knowledge to the student network. The steps are:

[0024] A1: Identify and select key anchor points in the training path of the teacher network, which represent the daily peaks and valleys of power load, emergency response to unexpected events, and seasonal load changes;

[0025] A2: At each anchor point, the output of the teacher network is divided into local feature blocks and global feature blocks. The local feature blocks capture the detailed information of the teacher network, and the global feature blocks are used to transfer the global mode of the model.

[0026] A3: By formula The local feature blocks and global feature blocks at different anchor points are combined, and the weights of each anchor point and block are adjusted according to the importance of the block and the learning needs of the student network, where F fused represents the weighted fusion result of local and global feature blocks, N is the total number of anchor points, and Respectively represent the local feature block and the global feature block at the i-th anchor point, and is the weight of the local feature block and the global feature block at the i-th anchor point, and the weight and Adjust to the learning needs of the student network;

[0027] A4: During the student network training process, the performance of the student network is dynamically evaluated and the learning priority of the functional blocks at different anchor points is adjusted so that the student network can acquire knowledge from the optimal anchor points and functional blocks at each learning stage.

[0028] Optionally, when the embedded physical laws are passed to the student network as guidance information, the physical law information is passed through the softmax output of the teacher network, and the softmax output needs to be temperature adjusted.

[0029] Optionally, during the knowledge distillation process, the hyperparameters including the temperature parameter, the hard loss and the soft loss ratio are adjusted, and the steps of the strategy of gradually reducing the soft loss ratio are:

[0030] Step S401: At the beginning of knowledge distillation, set the temperature parameter T to make the softmax distribution of the teacher network output smoother. Then, set a composite loss function that includes hard loss and soft loss: L total =αL soft +βL hard , where α and β are the weights of soft loss and hard loss. In the initial stage, α is greater than β;

[0031] Step S402: Using the set temperature parameter T, the softmax output of the teacher network is smoothed, making it easier for the student network to capture the detailed output of the teacher network. The student network gradually adjusts its weight by imitating the output characteristics of the teacher network to reduce the difference between the output of the student network and the teacher network.

[0032] Step S403: As the training progresses, the weight α of the soft loss is gradually reduced, while the weight β of the hard loss is increased, gradually transitioning to let the student network pay more attention to the learning of the real label, and then adjusting the weight by linear attenuation, the expression is: Where t is the number of training iterations, T max is the maximum training time, r is the exponential decay rate;

[0033] Step S404: As the proportion of soft loss decreases, the temperature parameter T is gradually reduced so that the student network can learn the details in the output of the teacher network and gradually shift from overall distribution learning to capturing precise features.

[0034] Optionally, during the training process of the student network, combined with the progressive knowledge distillation method, the training objective is decomposed into multiple intermediate objectives by gradually projecting the supervision signal of the teacher network into the parameter space of the student network, and the steps of gradually guiding the parameter adjustment and optimization of the student network are:

[0035] Step S411: decompose the overall training goal into multiple sub-goals, each sub-goal corresponds to a different complexity and learning task, and then, at each stage, gradually project the supervision signal of the teacher network into the parameter space of the student network;

[0036] Step S412: According to the defined intermediate objectives, gradually optimize the parameters of the student network in each training stage;

[0037] Step S413: In the middle and late stages of training, gradually reduce the supervision signal of the teacher network, reduce the weight of the soft loss, and enhance the student network's reliance on the true label;

[0038] Step S414: At the beginning and end of each stage, the hyperparameters are dynamically adjusted so that the student network gradually gets rid of its dependence on the teacher network and turns to learning based on real data. Then, in the later stage of training, the weight α of the soft loss is reduced to close to 0, so that the student network relies on the hard loss β for optimization.

[0039] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0040] The present invention establishes multiple teacher networks to learn different physical laws respectively, and embeds physical constraints in each network, which significantly improves the learning ability and performance of the student network in complex power dispatching problems. In addition, only the teacher network that performs better than the student network in the task is selected for knowledge transfer, avoiding the interference of inefficient and useless information, thereby improving learning efficiency and result reliability. At the same time, physical laws are used as guiding information throughout the knowledge transfer process, ensuring that the student network is not only limited to learning data patterns, but also follows the physical constraints in the power system, thereby enhancing the interpretability and stability of the model.

[0041] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0043] In the figure:

[0044] Figure 1 Flowchart of the method for optimizing neural network models based on knowledge distillation technology.

[0045] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0046] The present invention will now be described in further detail with reference to the accompanying drawings.

[0047] See also Figure 1 As shown, in this embodiment, a method for optimizing a neural network model based on knowledge distillation technology is provided, including the following steps:

[0048] Step S1: First, multiple teacher networks are established, each focusing on a different scheduling task, and then trained independently and embedded with power balance and voltage stability physical constraints;

[0049] Step S2: Evaluate the performance of all teacher networks and select the teacher networks that perform better than the student networks on specific scheduling tasks to ensure that knowledge transfer is only carried out from the teacher networks with better performance;

[0050] Step S3: During the knowledge transfer process, the output of the teacher network is temperature-conditioned and then passed to the student network, ensuring that the student network not only learns the data pattern but also follows the physical laws of the power system;

[0051] Step S4: During the knowledge distillation process, the hyperparameters including the temperature parameter, the hard loss and the soft loss ratio are adjusted, and the strategy of gradually reducing the soft loss ratio is adopted. During the training process of the student network, the supervision signal of the teacher network is gradually projected into the parameter space of the student network in combination with the progressive knowledge distillation method, and the training objective is decomposed into multiple intermediate objectives, which gradually guides the parameter adjustment and optimization of the student network.

[0052] The present invention establishes multiple teacher networks to learn different physical laws respectively, and embeds physical constraints in each network, which significantly improves the learning ability and performance of the student network in complex power dispatching problems. In addition, only the teacher network that performs better than the student network in the task is selected for knowledge transfer, avoiding the interference of inefficient and useless information, thereby improving learning efficiency and result reliability. At the same time, physical laws are used as guiding information throughout the knowledge transfer process, ensuring that the student network is not only limited to learning data patterns, but also follows the physical constraints in the power system, thereby enhancing the interpretability and stability of the model.

[0053] In this embodiment, multiple teacher network models are established and trained to learn different physical laws respectively, and each teacher network is trained independently and embedded with physical constraints. Then, the steps of establishing a student network model to be optimized are as follows:

[0054] Step S101: Select a convolutional neural network as the neural network architecture, then set initial weights, bias values, and learning rates for each teacher network, and then independently train multiple teacher networks to optimize their loss functions;

[0055] Step S102: In the training of the teacher network, a loss term related to physical laws is introduced on the basis of the original loss function and the physical law loss is combined with the conventional loss to form a composite loss function, which is expressed as: L total =L task +λ∑(P pred -P true ) 2 , where L task is the task loss, L phys is the physical law loss, λ is the coefficient that controls the weight of the two, P pred is the physical quantity predicted by the model, P true It is a real physical quantity, ensuring the conservation relationship in the output;

[0056] Step S103: By formula The power conservation and voltage stability are embedded in the network’s loss function, and the composite loss function containing physical constraints is used to independently train the teacher network so that it can follow the physical laws of the power system while learning the data pattern. j | is the voltage at node j, Vmin and V max are the lower and upper limits of voltage, and γ is the weight factor; the neural network model is used to optimize daily load forecasting and emergency response decisions. To this end, multiple teacher networks are established, each of which learns different physical laws, such as power conservation and voltage stability.

[0057] Step S104: During and after the training process, the model output is verified to ensure that it meets the physical constraints of power balance and voltage stability.

[0058] In power system dispatching, in order to improve the accuracy of daily load forecasting and emergency response, it is necessary to establish multiple teacher networks, each of which learns and follows different physical laws, such as power conservation and voltage stability.

[0059] In the implementation of step S102, it is assumed that there is a teacher network for load forecasting, and the original loss function is the mean square error (MSE), which is used to measure the difference between the predicted value and the actual value. In order to guide the network to follow the physical laws during the learning process, physical loss terms such as power conservation loss are introduced on this basis. The physical law loss is combined with the task loss to form a composite loss function, so that the network not only considers the accuracy of the data when predicting, but also maintains the power conservation requirement, thereby ensuring the physical rationality of the output.

[0060] Step S103 involves embedding power conservation and voltage stability into the network's loss function through formulas during training. For example, in a node in the power system, the network considers the node's power generation, load power, and power flow between it and other nodes. The goal of the network is to ensure that the output satisfies power conservation during training, that is, the balance between power generation and load. At the same time, the network checks whether the node voltage is within the allowable upper and lower limits to ensure voltage stability. The introduction of these physical constraints allows the network to not only optimize the task itself, but also follow the actual laws of the power system.

[0061] It should be noted that by adopting convolutional neural networks as the basic architecture and setting initial parameters for each teacher network, the flexibility and scalability of the neural network model are ensured. During the training process of the teacher network, the physical laws are embedded in the loss function, so that the loss function can not only optimize the task-related prediction error, but also constrain the model output to follow the physical laws, thereby enhancing the interpretability and physical consistency of the model. By introducing a composite loss function, combining task loss and physical law loss, the model can simultaneously learn data patterns and physical constraints to ensure that the output results conform to the actual physical conservation relationship. In addition, the physical phenomena are converted into linear functions using theoretical models and embedded into neural networks, so that the model can better adapt to problem scenarios with physical backgrounds, effectively improving the stability, accuracy and generalization ability of the model when dealing with physical problems.

[0062] In this embodiment, the performance-aware mutual knowledge distillation method is used to evaluate the performance of multiple teacher networks and only select the teacher network with better performance than the student network to transfer knowledge. The steps are:

[0063] Step S201: Evaluate the performance of all teacher networks and student networks on the same data set, and use predefined performance indicators to calculate the performance of each network on the task. At this time, evaluate the gap between the initial performance of the student network and the output of multiple teacher networks. Then, based on the results of the performance evaluation, select the teacher network with better performance than the student network.

[0064] Step S202: Among the selected high-quality teacher networks, the knowledge of the teacher network is transferred to the student network through knowledge distillation. The softmax output of the teacher network is used as the learning target of the student network. There are multiple excellent teacher networks. The outputs of these teacher networks are integrated through weighted fusion. The expression is: Among them, P fused is the softmax output after weighted fusion of multiple teacher networks, N is the number of teacher networks, represents the softmax output of the i-th teacher network, w i is the weight of the i-th teacher network, satisfying

[0065] Step S203: During the training process, the performance of the student network is regularly re-evaluated and compared with the teacher network. As the performance of the student network improves, some teacher networks no longer perform better than the student network. The selection of teacher networks is dynamically updated and teacher networks that are no longer better than the student network are eliminated.

[0066] Step S204: As the performance of the student network gradually surpasses that of the teacher network, the weight of the soft loss is lowered, the hard loss ratio of the student network to the true label is increased, and the independent learning ability of the student network is gradually enhanced, so that it can eventually perform tasks without relying on the teacher network.

[0067] It should be noted that the learning efficiency and independence of the student network are improved by gradually screening and dynamically adjusting the teacher network. First, by evaluating the performance of the teacher network and the student network, the teacher network with better performance than the student network is selected to ensure that the student network only obtains effective knowledge from the teacher network with better performance and avoids the interference of invalid information. In the process of knowledge distillation, the outputs of multiple teacher networks are weighted fused to ensure that the student network can learn from the advantages of different teacher networks and improve the quality of learning. Secondly, during the training of the student network, as its performance improves, the teacher network that is no longer better than the student network is regularly re-evaluated and eliminated, and the selection of the teacher network is dynamically updated to ensure that the knowledge distillation process remains efficient. Finally, by gradually reducing the weight of the soft loss and enhancing the hard loss dependence of the student network on the true label, the independent learning ability of the student network is gradually improved, so that it can eventually perform tasks without relying on the teacher network.

[0068] In this embodiment, step S2 selects multiple key anchor points in the teacher network training path during data transmission, and performs block processing on the output at the anchor points to transmit local and global knowledge to the student network. The steps are:

[0069] A1: Identify and select key anchor points in the training path of the teacher network, which represent the daily peaks and valleys of power load, emergency response to unexpected events, and seasonal load changes;

[0070] A2: At each selected anchor point, the output of the teacher network is divided into local feature blocks and global feature blocks. The local feature blocks are used to capture detailed information such as short-term load changes and rapid response capabilities, while the global feature blocks provide information on the overall trend and long-term stability of the power system to help the student network fully understand the system behavior;

[0071] A3: By formula: The local feature blocks and global feature blocks at different anchor points are combined, and the weights of each anchor point and block are adjusted according to the importance of the block and the learning needs of the student network, where F fused represents the weighted fusion result of local and global feature blocks, N is the total number of anchor points, and Respectively represent the local feature block and the global feature block at the i-th anchor point, and is the weight of the local feature block and the global feature block at the i-th anchor point, and the weight and Adjust to the learning needs of the student network;

[0072] A4: During the training of the student network, the learning effect of different anchor points and feature blocks is regularly evaluated, and the learning priority is dynamically adjusted according to the evaluation results. This allows the student network to optimize its learning path at different stages of training, ensuring that it focuses on global features in the early stages and details in the later stages, so as to improve its ability and stability in coping with complex scheduling tasks.

[0073] It should be noted that by dynamically evaluating and adjusting the weights of multiple training stages and functional blocks of the teacher network, the learning efficiency and accuracy of the student network at different learning stages are improved. First, during the training process of the teacher network, the training path is recorded and multiple anchor points representing important states are selected, so that the student network can not only learn the final performance of the teacher network, but also absorb important information of the teacher network at each stage of the training process. Secondly, by dividing the output at each anchor point into local feature blocks and global feature blocks, the student network can capture both detailed information and global patterns to ensure the comprehensiveness of knowledge transfer. The weighted fusion mechanism dynamically adjusts the weights of local and global feature blocks according to the learning needs of the student network, and flexibly adapts to the learning goals of the student network at different stages. Finally, during the training process of the student network, its performance is continuously and dynamically evaluated, and the learning priority of each anchor point and functional block is adjusted, so that the student network can focus on acquiring knowledge from the optimal anchor point and functional block at each stage.

[0074] In this embodiment, when the embedded physical law is passed to the student network as guidance information, the physical law information is passed through the softmax output of the teacher network, and the softmax output needs to be temperature adjusted.

[0075] In this embodiment, in the knowledge distillation process, the steps of adjusting the hyper parameters including the temperature parameter, the hard loss and the soft loss ratio, and gradually reducing the soft loss ratio are as follows:

[0076] Step S401: At the beginning of knowledge distillation, set the temperature parameter T to make the softmax distribution of the teacher network output smoother. Then, set a composite loss function that includes hard loss and soft loss: L total =αL soft +βL hard , where α and β are the weights of soft loss and hard loss. In the initial stage, α is greater than β;

[0077] Step S402: Using the set temperature parameter T, the softmax output of the teacher network is smoothed, making it easier for the student network to capture the detailed output of the teacher network. The student network gradually adjusts its weight by imitating the output characteristics of the teacher network to reduce the difference between the output of the student network and the teacher network.

[0078] Step S403: As the training progresses, the weight α of the soft loss is gradually reduced, while the weight β of the hard loss is increased, gradually transitioning to let the student network pay more attention to the learning of the real label, and then adjusting the weight by linear attenuation, the expression is: Where t is the number of training iterations, T max is the maximum training time, r is the exponential decay rate;

[0079] Step S404: As the proportion of soft loss decreases, the temperature parameter T is gradually reduced so that the student network can learn the details in the output of the teacher network and gradually shift from overall distribution learning to capturing precise features.

[0080] It should be noted that by gradually adjusting the temperature parameter and the weight of the loss function, a smooth transition of the student network from imitating the teacher network to independent learning is achieved. In the early stage of knowledge distillation, the temperature parameter T is set to smooth the s output of the teacher network, so that the student network can more easily capture the details of the teacher network, and a composite loss function dominated by soft loss is set to ensure that the student network can effectively learn the knowledge of the teacher network. By setting the initial soft loss ratio higher than the hard loss, the student network focuses on imitating the output characteristics of the teacher network, and gradually adjusts its weights to reduce the difference with the teacher network. As the training progresses, the weight of the soft loss is gradually reduced, and the proportion of the hard loss is increased, so that the student network gradually turns to learning the real label. The weight is adjusted by linear attenuation so that the student network can learn more independently, and at the same time, the temperature parameter T is gradually reduced, so that the student network turns to capturing more refined feature information.

[0081] In this embodiment, during the training process of the student network, the progressive knowledge distillation method is combined to gradually project the supervision signal of the teacher network into the parameter space of the student network, decompose the training target into multiple intermediate targets, and gradually guide the steps of adjusting and optimizing the parameters of the student network:

[0082] Step S411: decompose the overall training goal into multiple sub-goals, each sub-goal corresponds to a different complexity and learning task, and then, at each stage, gradually project the supervision signal of the teacher network into the parameter space of the student network;

[0083] Step S412: According to the defined intermediate objectives, gradually optimize the parameters of the student network in each training stage;

[0084] Step S413: In the middle and late stages of training, gradually reduce the supervision signal of the teacher network, reduce the weight of the soft loss, and enhance the student network's reliance on the true label;

[0085] Step S414: At the beginning and end of each stage, the hyperparameters are dynamically adjusted so that the student network gradually gets rid of its dependence on the teacher network and turns to learning based on real data. Then, in the later stage of training, the weight α of the soft loss is reduced to close to 0, so that the student network relies on the hard loss β for optimization.

[0086] It should be noted that by decomposing the overall training goal into multiple sub-goals and gradually guiding the student network to learn at different stages, the learning efficiency and task adaptability of the student network are significantly improved. First, the training process is decomposed into multiple sub-goals, so that the student network can gradually complete learning tasks of different complexity from easy to difficult, ensuring that it can efficiently absorb the knowledge of the teacher network at each stage. By gradually projecting the supervision signal of the teacher network into the parameter space of the student network at each stage, it is ensured that the student network can fully rely on the guidance information of the teacher network to optimize the parameters in the early stage. As the training progresses, the supervision signal of the teacher network is gradually reduced in the middle and late stages, and the weight of the soft loss is reduced, so that the student network gradually relies on the real label, thereby enhancing its independent learning ability. By dynamically adjusting the hyperparameters at each stage, it is ensured that the student network smoothly transitions from relying on the teacher network to learning based on real data. Finally, by gradually reducing the weight of the soft loss to near zero, the student network mainly relies on the hard loss for optimization, enhancing its ability to perform tasks without the support of the teacher network, and improving the generalization and autonomous learning ability of the model.

[0087] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A method for optimizing a neural network model based on knowledge distillation technology, characterized in that: The following steps are involved: Step S1: First, multiple teacher networks are established, each focusing on a different scheduling task, and then trained independently and embedded with power balance and voltage stability physical constraints; Step S2: Evaluate the performance of all teacher networks and select the teacher networks that perform better than the student networks on specific scheduling tasks to ensure that knowledge transfer is only carried out from the teacher networks with better performance; Step S3: During the knowledge transfer process, the output of the teacher network is temperature-conditioned and then passed to the student network, ensuring that the student network not only learns the data pattern but also follows the physical laws of the power system; Step S4: During the knowledge distillation process, the hyperparameters including the temperature parameter, the hard loss and the soft loss ratio are adjusted, and the strategy of gradually reducing the soft loss ratio is adopted. During the training process of the student network, the supervision signal of the teacher network is gradually projected into the parameter space of the student network in combination with the progressive knowledge distillation method, and the training objective is decomposed into multiple intermediate objectives, which gradually guides the parameter adjustment and optimization of the student network.

2. According to claim 1, a method for optimizing a neural network model based on knowledge distillation technology is characterized in that: Establish and train multiple teacher network models, train them to learn different physical laws respectively, and each teacher network is trained independently and embedded with physical constraints. Then, the steps to establish a student network model to be optimized are: Step S101: Select a convolutional neural network as the neural network architecture, then set initial weights, bias values, and learning rates for each teacher network, and then independently train multiple teacher networks to optimize their loss functions; Step S102: In the training of the teacher network, a loss term related to physical laws is introduced on the basis of the original loss function and the physical law loss is combined with the conventional loss to form a composite loss function, which is expressed as: L total =L task +λ∑(P pred -P true ) 2 , where L task is the task loss, L phys is the physical law loss, λ is the coefficient that controls the weight of the two, P pred is the physical quantity predicted by the model, P true It is a real physical quantity, ensuring the conservation relationship in the output; Step S103: By formula The power conservation and voltage stability are embedded in the network’s loss function, and the composite loss function containing physical constraints is used to independently train the teacher network so that it can follow the physical laws of the power system while learning the data pattern. j | is the voltage at node j, V min and V max are the lower and upper limits of the voltage, and γ is the weighting factor; Step S104: During and after the training process, the model output is verified to ensure that it meets the physical constraints of power balance and voltage stability.

3. The method for optimizing a neural network model based on knowledge distillation technology according to claim 1, characterized in that: The performance-aware mutual knowledge distillation method is used to evaluate the performance of multiple teacher networks and only select the teacher network with better performance than the student network to transfer knowledge. The steps are: Step S201: Evaluate the performance of all teacher networks and student networks on the same data set, and use predefined performance indicators to calculate the performance of each network on the task. At this time, evaluate the gap between the initial performance of the student network and the output of multiple teacher networks. Then, based on the results of the performance evaluation, select the teacher network with better performance than the student network. Step S202: Among the selected high-quality teacher networks, the knowledge of the teacher network is transferred to the student network through knowledge distillation. The softmax output of the teacher network is used as the learning target of the student network. There are multiple excellent teacher networks. The outputs of these teacher networks are integrated through weighted fusion. The expression is: Among them, P fused is the softmax output after weighted fusion of multiple teacher networks, N is the number of teacher networks, represents the softmax output of the i-th teacher network, w i is the weight of the i-th teacher network, satisfying Step S203: During the training process, the performance of the student network is regularly re-evaluated and compared with the teacher network. As the performance of the student network improves, some teacher networks no longer perform better than the student network. The selection of teacher networks is dynamically updated and teacher networks that are no longer better than the student network are eliminated. Step S204: As the performance of the student network gradually surpasses that of the teacher network, the weight of the soft loss is lowered, the hard loss ratio of the student network to the true label is increased, and the independent learning ability of the student network is gradually enhanced, so that it can eventually perform tasks without relying on the teacher network.

4. The method for optimizing a neural network model based on knowledge distillation technology according to claim 1, characterized in that: Step S2 transfers local and global knowledge to the student network by selecting multiple key anchor points in the teacher network training path and processing the output in blocks at the anchor points during the data transfer process. The steps are: A1: Identify and select key anchor points in the training path of the teacher network, which represent the daily peaks and valleys of power load, emergency response to unexpected events, and seasonal load changes; A2: At each anchor point, the output of the teacher network is divided into local feature blocks and global feature blocks. The local feature blocks capture the detailed information of the teacher network, and the global feature blocks are used to transfer the global mode of the model. A3: By formula The local feature blocks and global feature blocks at different anchor points are combined, and the weights of each anchor point and block are adjusted according to the importance of the block and the learning needs of the student network, where F fused represents the weighted fusion result of local and global feature blocks, N is the total number of anchor points, and Respectively represent the local feature block and the global feature block at the i-th anchor point, and is the weight of the local feature block and the global feature block at the i-th anchor point, and the weight and Adjust to the learning needs of the student network; A4: During the student network training process, the performance of the student network is dynamically evaluated and the learning priority of the functional blocks at different anchor points is adjusted so that the student network can acquire knowledge from the optimal anchor points and functional blocks at each learning stage.

5. The method for optimizing a neural network model based on knowledge distillation technology according to claim 1, characterized in that: When the embedded physical laws are passed to the student network as guidance information, the physical law information is transmitted through the softmax output of the teacher network, and the softmax output needs to be temperature adjusted.

6. The method for optimizing a neural network model based on knowledge distillation technology according to claim 1, characterized in that: In the process of knowledge distillation, the hyperparameters including temperature parameters, hard loss and soft loss ratio are adjusted, and the steps of the strategy of gradually reducing the soft loss ratio are as follows: Step S401: At the beginning of knowledge distillation, set the temperature parameter T to make the softmax distribution of the teacher network output smoother. Then, set a composite loss function that includes hard loss and soft loss: L total =αL soft +βL hard , where α and β are the weights of soft loss and hard loss. In the initial stage, α is greater than β; Step S402: Using the set temperature parameter T, the softmax output of the teacher network is smoothed, making it easier for the student network to capture the detailed output of the teacher network. The student network gradually adjusts its weight by imitating the output characteristics of the teacher network to reduce the difference between the output of the student network and the teacher network. Step S403: As the training progresses, the weight α of the soft loss is gradually reduced, while the weight β of the hard loss is increased, gradually transitioning to let the student network pay more attention to the learning of the real label, and then adjusting the weight by linear attenuation, the expression is: Where t is the number of training iterations, T max is the maximum training time, r is the exponential decay rate; Step S404: As the proportion of soft loss decreases, the temperature parameter T is gradually reduced so that the student network can learn the details in the output of the teacher network and gradually shift from overall distribution learning to capturing precise features.

7. The method for optimizing a neural network model based on knowledge distillation technology according to claim 1, characterized in that: In the training process of the student network, combined with the progressive knowledge distillation method, the training objective is decomposed into multiple intermediate objectives by gradually projecting the supervision signal of the teacher network into the parameter space of the student network, and the steps of gradually guiding the parameter adjustment and optimization of the student network are as follows: Step S411: decompose the overall training goal into multiple sub-goals, each sub-goal corresponds to a different complexity and learning task, and then, at each stage, gradually project the supervision signal of the teacher network into the parameter space of the student network; Step S412: According to the defined intermediate objectives, gradually optimize the parameters of the student network in each training stage; Step S413: In the middle and late stages of training, gradually reduce the supervision signal of the teacher network, reduce the weight of the soft loss, and enhance the student network's reliance on the true label; Step S414: At the beginning and end of each stage, the hyperparameters are dynamically adjusted so that the student network gradually gets rid of its dependence on the teacher network and turns to learning based on real data. Then, in the later stage of training, the weight α of the soft loss is reduced to close to 0, so that the student network relies on the hard loss β for optimization.

Images