Metareinforcement learning model migration training method and device based on topological transformation

Abstract

The invention discloses a meta reinforcement learning model migration training method and device based on topological transformation, and relates to the technical field of machine learning. Obtaining power data of a plurality of regions, and constructing a region data set; performing data feature comparison on the region data set and a model data set of a preset training model, and determining an updating strategy of the preset training model according to data features; and migrating data in the regional data set into the model data set, and executing an updating operation according to the updating strategy to obtain a target model. According to the method, the regional data set is constructed by collecting the multi-regional power data, the regional data set is subjected to data feature comparison with the model data set of the preset training model to determine the model updating strategy, and the migration training method is effectively applied to model updating. According to the method, the training process is accelerated, the model performance is improved, and the computing resource demand is reduced, so that an efficient and economical model updating mode is realized, and the adaptability and accuracy of the model in power fault positioning are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a meta-reinforcement learning model migration training method and device based on topological transformation. Background Art

[0002] The transfer training method of the meta-reinforcement learning model plays a significant role in the current data-driven and intelligent context. By learning and transferring knowledge from multiple tasks, the model can quickly adapt to the new environment, optimize the decision-making process, and efficiently solve problems and optimize strategies in scientific research, industry, medicine, finance and other fields. This technology provides strong technical support for the development and application of the model.

[0003] Patent CN115800272A, the present invention provides a disclosed power grid fault analysis method, system, terminal and medium based on topology identification; after building the power grid topology model, the risk value is predicted based on the weight analysis of abnormal electrical quantities, and the associated topology nodes are extracted and their associated risk values ​​are calculated. The associated risk value and the predicted risk value are superimposed to obtain the cumulative risk value, and the node with the largest difference between the cumulative risk value and the threshold is selected as the predicted fault node, and the screening is continued until the cumulative risk value is lower than the threshold. The models constructed in the prior art are often targeted at specific areas, and the models do not have adaptability after they are established, resulting in the need to rebuild the corresponding model for each scenario, which increases the consumed resources. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the model does not have the adaptability after it is established, resulting in the need to rebuild the corresponding model for each scene, and to propose a meta-reinforcement learning model transfer training method and device based on topological transformation.

[0005] In a first aspect of the present invention, a meta-reinforcement learning model transfer training method based on topological transformation is first proposed, and the method comprises:

[0006] Acquire power data of multiple regions and construct regional data sets; the regional data sets include: training data sets, verification data sets, and test data sets; the power data include: power grid topology and power data of various devices;

[0007] Comparing data features of the regional data set with a model data set of the preset training model, and determining an update strategy for the preset training model according to the data features;

[0008] The data in the regional data set is migrated to the model data set, and an update operation is performed according to the update strategy to obtain a target model.

[0009] Optionally, comparing data features of the regional data set with a model data set of the preset training model includes:

[0010] Extracting key features of the regional data set and the model data set; the key features include: key node features in the power grid topology and power data features of the equipment;

[0011] Calculating an absolute difference between the regional data set and the key features of the preset training model, and if the absolute difference is less than a preset threshold, determining that the regional data set has a high similarity with the preset training model;

[0012] The data volume of the regional data set is compared with a data volume threshold, and if the data volume of the regional data set is less than the data volume threshold, it is determined that the data volume of the regional data set is small.

[0013] Optionally, determining an update strategy for the preset training model according to the data feature includes:

[0014] If the data volume of the regional data set is small and the similarity is high, determining that the update strategy of the preset training model is the first update mode;

[0015] Freeze the model parameters of the convolution layer and the pooling layer of the preset training model, unfreeze the model parameters of the fully connected layer and the classification layer of the preset training model, input the data of the regional data set into the preset training model for model training, and update the parameters of the fully connected layer and the classification layer to obtain the target model.

[0016] Optionally, determining an update strategy for the preset training model according to the data feature includes:

[0017] If the data volume of the regional data set is small and the similarity is low, determining that the update strategy of the preset training model is the second update mode;

[0018] The model parameters of the fully connected layer of the preset model are unfrozen, the regional data set is input into the preset training model for model training, and the parameters of the fully connected layer are updated to obtain the target model.

[0019] Optionally, determining an update strategy for the preset training model according to the data feature includes:

[0020] If the data volume of the regional data set is large and the similarity is high, determining that the update strategy of the preset training model is the third update mode;

[0021] The model parameters of each layer in the preset training model are unfrozen in a step-by-step incremental mode, and the data of the regional data set is input into the preset training model for incremental training until the parameters of each layer in the preset training model are updated to obtain the target model.

[0022] Optionally, determining an update strategy for the preset training model according to the data feature includes:

[0023] If the data volume of the regional data set is large and the similarity is low, determining that the update strategy of the preset training model is the fourth update mode;

[0024] The parameters of the preset training model are initialized, the data in the regional data set is added to the model data set of the preset training model for model training, and the model parameters of the preset training model are updated to obtain the target model.

[0025] In a second aspect of the present invention, a meta-reinforcement learning model migration training device based on topological transformation is proposed, comprising: the device comprises: a data set construction module, a similarity comparison module and a model update module:

[0026] The data set construction module is used to obtain power data of multiple regions and construct regional data sets; the regional data sets include: training data sets, verification data sets, and test data sets; the power data include: power grid topology and power data of various devices;

[0027] The similarity comparison module is used to compare data features of the regional data set with the model data set of the preset training model, and determine an update strategy for the preset training model according to the data features;

[0028] The model updating module is used to migrate the data in the regional data set to the model data set, and perform an updating operation according to the updating strategy to obtain a target model.

[0029] Beneficial effects of the present invention:

[0030] The present invention proposes a meta-reinforcement learning model migration training method based on topological transformation, which constructs a regional data set by acquiring power data from multiple regions; compares data features between the regional data set and a model data set of a preset training model, and determines an update strategy for the preset training model based on the data features; migrates data in the regional data set to the model data set, and performs an update operation based on the update strategy to obtain a target model. By collecting power data from multiple regions to construct a regional data set, and comparing data features with the model data set of a preset training model to determine a model update strategy, the migration training method is effectively applied to model updates. This method not only accelerates the training process and improves model performance, but also reduces computing resource requirements, thereby achieving an efficient and economical model update method and enhancing the adaptability and accuracy of the model in power fault location. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] Figure 1 A flowchart of a meta-reinforcement learning model migration training method based on topological transformation is provided for an embodiment of the present invention;

[0033] Figure 2 A structural schematic diagram of another meta-reinforcement learning model transfer training device based on topological transformation is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the present invention, the description of "first", "second", etc. is only used for descriptive purposes, and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0035] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0036] The embodiment of the present invention provides a meta-reinforcement learning model transfer training method based on topological transformation. Figure 1 , Figure 1 A flowchart of a meta-reinforcement learning model migration training method based on topological transformation provided by an embodiment of the present invention. The method comprises the following steps:

[0037] S101, obtaining power data of multiple regions and constructing regional data sets.

[0038] S102, comparing data features of the regional data set with the model data set of the preset training model, and determining an update strategy for the preset training model according to the data features.

[0039] S103, migrating the data in the regional data set to the model data set, and performing an update operation according to the update strategy to obtain the target model.

[0040] The regional data sets include: training data sets, verification data sets, and test data sets; the power data include: power grid topology and power data of various equipment.

[0041] Based on a meta-reinforcement learning model migration training method based on topological transformation provided by an embodiment of the present invention, a regional data set is constructed by collecting multi-regional power data, and data features are compared with the model data set of a preset training model to determine the model update strategy. The migration training method is effectively applied to model update. This method not only speeds up the training process and improves model performance, but also reduces computing resource requirements, thereby achieving an efficient and economical model update method and enhancing the adaptability and accuracy of the model in power fault location.

[0042] In one implementation method, the grid topology in the power data changes due to upgrades or other reasons, so the model needs to be upgraded according to the current situation. Model training through migration training can greatly reduce the amount of training, improve resource utilization, and reduce unnecessary costs.

[0043] In one implementation, the update strategy includes: a first update mode, a second update mode, a third update mode and a fourth update mode; wherein, if the data volume of the regional data set is small and the similarity is high, the mode is determined to be the first update mode, if the data volume of the regional data set is small and the similarity is low, the mode is determined to be the second update mode, if the data volume of the regional data set is large and the similarity is high, the mode is determined to be the third update mode, and if the data volume of the regional data set is large and the similarity is low, the mode is determined to be the fourth update mode; the first update mode can reduce unnecessary training costs and enable the model to adapt to the new environment; the second update mode can enable the model to adapt to the new environment using a small amount of training costs; the third update mode adapts the model to the new data set through step-by-step training, and can also maintain the performance of the model on the original data set to achieve better migration effects; the fourth update mode is because the new data set is large and the differences are also large, and retraining the model can make full use of these rich data resources, thereby further improving the performance and accuracy of the model.

[0044] In one implementation, by comparing the data features of the regional data set with the model data set of the preset training model and determining the model update strategy accordingly, the model can be efficiently adjusted to better determine the update mode, thereby improving the accuracy of the model in the new environment without sacrificing the original performance, achieving more accurate power grid fault location, and saving time and computing resources.

[0045] In one embodiment, step S102 further includes:

[0046] Extract key features of the regional data set and the model data set; the key features include: key node features in the power grid topology and power data features of the equipment.

[0047] The absolute difference between the key features of the regional dataset and the preset training model is calculated. If the absolute difference is less than the preset threshold, it is determined that the regional dataset has a high similarity with the preset training model.

[0048] The data volume of the regional data set is compared with the data volume threshold. If the data volume of the regional data set is less than the data volume threshold, it is determined that the data volume of the regional data set is small.

[0049] In one implementation, the absolute difference between the key features of the regional dataset and the model dataset is calculated by cosine similarity, and this method is used to judge the similarity between the two datasets, which can provide a quantitative indicator to evaluate the similarity between the two datasets. The advantage of this method is its intuitiveness and accuracy. By calculating the absolute difference, the degree of difference between the two datasets in key features can be obtained, so as to quickly determine the update strategy of the model. Using a preset threshold as the judgment standard can ensure that only when the difference reaches a certain level, the similarity difference between the two datasets will be judged to be large.

[0050] In one implementation, key features of the regional dataset and the model dataset are extracted, especially key node features in the power grid topology and power data features of the equipment, to ensure the accuracy and effectiveness of the model update.

[0051] In one embodiment, determining an update strategy for a preset training model according to data features further includes:

[0052] If the data volume of the regional data set is small and the similarity is high, the update strategy of the preset training model is determined to be the first update mode.

[0053] Freeze the model parameters of the convolution layer and pooling layer of the preset training model, unfreeze the model parameters of the fully connected layer and classification layer of the preset training model, input the data of the regional dataset into the preset training model for model training, and update the parameters of the fully connected layer and classification layer to obtain the target model.

[0054] In one implementation, freezing is to lock the model parameters of the layers in the model and prohibit the model parameters of each layer from being updated; unfreezing is to unlock the model parameters of the layers in the model and allow the model parameters of each layer to be updated.

[0055] In one implementation, when the first update mode is used to update the model, the transfer training process brings significant benefits. Because the regional data set has a small amount of data but a high degree of similarity, the knowledge in the pre-trained preset training model, that is, the general feature representation that has been learned, is utilized through transfer training. This feature representation may still be effective on new tasks, thereby reducing the large amount of time and computing resources required to train the model from scratch. By updating the parameters of the fully connected layer and classification layer in the model, transfer training can retain the feature extraction capabilities learned by the preset training model on similar tasks, while enabling the model to adapt to specific tasks of the new data set. This adaptability enables the model to converge quickly on the new data set, improves training efficiency, and often achieves better performance and reduces resource waste.

[0056] In one implementation, when only the parameters of the fully connected layer and the classification layer are updated, in a deep neural network, the convolution layer and the pooling layer are mainly responsible for learning the low-level feature representation of the data, such as edges, textures, etc., which are often common between multiple tasks. The fully connected layer and the classification layer are more focused on learning the high-level feature representation and decision boundaries of the data, which are closely related to specific tasks. In the first update mode, since the regional data set has a high similarity with the preset training model, it can be considered that the convolution layer and the pooling layer have learned a good enough feature representation, so no update is required. The fully connected layer and the classification layer need to be fine-tuned according to the new data set to adapt to the specific requirements of the new task. The advantage of this choice is that it can improve the adaptability and accuracy of the model on the new data set while maintaining the performance of the model on similar tasks. By updating only the parameters of the fully connected layer and the classification layer, the model can quickly and effectively complete the migration training, save computing resources, and obtain better model performance.

[0057] In one embodiment, determining an update strategy for a preset training model according to data features further includes:

[0058] If the data volume of the regional data set is small and the similarity is low, the update strategy of the preset training model is determined to be the second update mode.

[0059] Unfreeze the model parameters of the fully connected layer of the preset model, input the regional dataset into the preset training model for model training, and update the parameters of the fully connected layer to obtain the target model.

[0060] In one implementation, when the second update mode is used to update the model, since the regional data set has a small amount of data but low similarity, training a new model from scratch is not only inefficient, but may also be difficult to capture enough feature information to adapt to the new task. Transfer training allows experimenters to use the knowledge learned by the preset training model on a large amount of similar data, especially its capabilities in feature extraction and representation learning. By unfreezing and updating the fully connected layers of the preset model, the model can perform targeted learning for the new data set while maintaining these common feature representations. The benefit of this training process is that it can minimize the consumption of computing resources and improve training efficiency while ensuring the adaptability of the model to the new data set.

[0061] In one implementation, in the second update mode, the fully connected layer plays an important role in the neural network to map the learned features to the final output. It is responsible for making the final decision or prediction based on the input feature representation. Since the regional dataset has a low similarity with the preset training model, the decision boundary and feature combination learned by the fully connected layer may need to be adjusted to adapt to the characteristics of the new dataset. Therefore, by only updating the parameters of the fully connected layer, targeted fine-tuning can be performed for the new dataset while keeping the model's feature extraction capabilities unchanged. The benefit of this choice is that it can avoid large-scale modifications to the overall structure of the model while ensuring the adaptability of the model to the new dataset, thereby maintaining the stability and reliability of the model.

[0062] In one embodiment, determining an update strategy for a preset training model according to data features further includes:

[0063] If the data volume of the regional data set is large and the similarity is high, the update strategy of the preset training model is determined to be the third update mode.

[0064] The model parameters of each layer in the preset training model are unfrozen in a step-by-step incremental mode, and the data of the regional data set are input into the preset training model for incremental training until the parameters of each layer in the preset training model are updated to obtain the target model.

[0065] In one implementation, in the third update mode, when the data set of the regional data set is large and the similarity is high, the parameters of each layer in the preset training model are unfrozen and updated in a step-by-step incremental mode for migration training, and the model is fine-tuned by making full use of a large amount of data to ensure that the model can achieve better performance on the new data set. By unfreezing the model layer step by step, the feature information of the new data set can be gradually integrated into the model, so that the model can gradually adapt to the distribution and pattern of the new data during the update process. This can not only improve the accuracy of the model on the new data set, but also maintain the performance of the model on the original data set, achieving better migration effects.

[0066] In one implementation, in the third update mode, the reason for selecting each layer for incremental update is that different network layers have different functions and importance in the model. By incrementally unfreezing and updating the parameters of each layer step by step, it can be ensured that the model can gradually absorb the information of new data during the update process, and better adapt to the new data set while maintaining the original feature extraction capability. This incremental update method can avoid the problems of model instability and performance degradation caused by unfreezing all layers at once. At the same time, the update order between each layer is also determined according to their role and importance in the model, so as to ensure that the model can gradually optimize its performance during the update process, and finally obtain a target model with excellent performance on the new data set. This method of selecting each layer for incremental update not only improves the training efficiency of the model, but also ensures the generalization ability and accuracy of the model on the new data set.

[0067] In one embodiment, determining an update strategy for a preset training model according to data features further includes:

[0068] If the data volume of the regional data set is large and the similarity is low, the update strategy of the preset training model is determined to be the fourth update mode.

[0069] The parameters of the preset training model are initialized, the data in the regional dataset is added to the model dataset of the preset training model for model training, and the model parameters of the preset training model are updated to obtain the target model.

[0070] In one implementation, in the fourth update mode, when the data set of the regional data set is large and the similarity is low, the parameters of the preset training model are initialized, and the data in the regional data set is added to the model data set for complete model training. This training process brings significant benefits. Due to the large data set, the model can learn more details and patterns from it, which helps to improve the generalization ability of the model on new tasks. Because the similarity of the data set is low, it is necessary to completely retrain the model to ensure that the model can adapt to the new data distribution and characteristics. This migration training process can make full use of the information of a large amount of new data, deeply adjust and optimize the model, and thus obtain a more accurate and more powerful target model.

[0071] In one implementation, in the fourth update mode, the reason for choosing to re-update the entire model is that when there are large differences in feature representation between the new data and the original data, simply fine-tuning some parameters of the model may not be able to fully adapt to the characteristics of the new data. At this time, reinitializing the parameters of the model and adding the new data to the model data set for complete model training can ensure that the model can learn the feature representation of the new data from scratch, so as to better adapt to the new task. The advantage of this way of re-updating the model is that it can break the original model's inherent cognition of data features and re-learn the inherent laws and patterns of the new data, thereby obtaining a more flexible and adaptable model. In addition, since the new data set is large, retraining the model can make full use of these rich data resources to further improve the performance and accuracy of the model.

[0072] Based on the same inventive concept, the embodiment of the present invention also provides a meta-reinforcement learning model transfer training device based on topological transformation. Figure 2 , Figure 2 A schematic diagram of the structure of a meta-reinforcement learning model migration training device based on topological transformation provided by an embodiment of the present invention includes: a data set construction module, a similarity comparison module and a model update module:

[0073] The data set construction module is used to obtain power data of multiple regions and construct regional data sets; the regional data sets include: training data sets, verification data sets, and test data sets; the power data include: power grid topology and power data of various equipment.

[0074] The similarity comparison module is used to compare the data features of the regional data set with the model data set of the preset training model, and determine the update strategy of the preset training model according to the data features.

[0075] The model update module is used to migrate the data in the regional dataset to the model dataset and perform the update operation according to the update strategy to obtain the target model.

[0076] Based on a meta-reinforcement learning model migration training device based on topological transformation provided by an embodiment of the present invention, a regional data set is constructed by collecting multi-regional power data, and data features are compared with the model data set of a preset training model to determine the model update strategy. The migration training method is effectively applied to model update. This method not only speeds up the training process and improves model performance, but also reduces computing resource requirements, thereby realizing an efficient and economical model update method and enhancing the adaptability and accuracy of the model in power fault location.

[0077] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A meta-reinforcement learning model transfer training method based on topological transformation, characterized in that: Obtaining a preset training model, wherein the preset training model is used to identify power grid faults in a target area, the method comprising: Obtain power data from multiple regions and build regional data sets; The regional data set includes: a training data set, a verification data set, and a test data set; the power data includes: power grid topology and power data of various devices; Comparing data features of the regional data set with a model data set of the preset training model, and determining an update strategy for the preset training model according to the data features; The data in the regional data set is migrated to the model data set, and an update operation is performed according to the update strategy to obtain a target model.

2. According to the topological transformation-based meta-reinforcement learning model transfer training method of claim 1, it is characterized in that: Comparing data features of the regional data set with the model data set of the preset training model includes: Extracting key features of the regional data set and the model data set; the key features include: key node features in the power grid topology and power data features of the equipment; Calculating an absolute difference between the regional data set and the key features of the preset training model, and if the absolute difference is less than a preset threshold, determining that the regional data set has a high similarity with the preset training model; The data volume of the regional data set is compared with a data volume threshold, and if the data volume of the regional data set is less than the data volume threshold, it is determined that the data volume of the regional data set is small.

3. According to the topological transformation-based meta-reinforcement learning model transfer training method of claim 2, it is characterized in that: Determining the update strategy of the preset training model according to the data characteristics includes: If the data volume of the regional data set is small and the similarity is high, determining that the update strategy of the preset training model is the first update mode; Freeze the model parameters of the convolution layer and the pooling layer of the preset training model, unfreeze the model parameters of the fully connected layer and the classification layer of the preset training model, input the data of the regional data set into the preset training model for model training, and update the parameters of the fully connected layer and the classification layer to obtain the target model.

4. The meta-reinforcement learning model transfer training method based on topological transformation according to claim 2 is characterized in that: Determining the update strategy of the preset training model according to the data characteristics includes: If the data volume of the regional data set is small and the similarity is low, determining that the update strategy of the preset training model is the second update mode; The model parameters of the fully connected layer of the preset model are unfrozen, the regional data set is input into the preset training model for model training, and the parameters of the fully connected layer are updated to obtain the target model.

5. According to a meta-reinforcement learning model transfer training method based on topological transformation according to claim 2, it is characterized in that: Determining the update strategy of the preset training model according to the data characteristics includes: If the data volume of the regional data set is large and the similarity is high, determining that the update strategy of the preset training model is the third update mode; The model parameters of each layer in the preset training model are unfrozen in a step-by-step incremental mode, and the data of the regional data set is input into the preset training model for incremental training until the parameters of each layer in the preset training model are updated to obtain the target model.

6. The meta-reinforcement learning model transfer training method based on topological transformation according to claim 2 is characterized in that: Determining the update strategy of the preset training model according to the data characteristics includes: If the data volume of the regional data set is large and the similarity is low, determining that the update strategy of the preset training model is the fourth update mode; The parameters of the preset training model are initialized, the data in the regional data set is added to the model data set of the preset training model for model training, and the model parameters of the preset training model are updated to obtain the target model.

7. A meta-reinforcement learning model transfer training device based on topological transformation, characterized in that: The device comprises: a data set construction module, a similarity comparison module and a model update module: The data set construction module is used to obtain power data of multiple regions and construct regional data sets; The regional data set includes: a training data set, a verification data set, and a test data set; the power data includes: power grid topology and power data of various devices; The similarity comparison module is used to compare data features of the regional data set with the model data set of the preset training model, and determine an update strategy for the preset training model according to the data features; The model updating module is used to migrate the data in the regional data set to the model data set, and perform an updating operation according to the updating strategy to obtain a target model.

Images