Method, device and equipment for training fault alarm handling scheme generation model and storage medium

Abstract

The application discloses a training method and device of a fault alarm treatment scheme generation model, an equipment and a storage medium. An initial preference model is trained through historical labeled data, a pre-trained preference model is obtained, and the pre-trained preference model is migrated to a pre-constructed reinforcement learning intelligent agent to obtain an initial reinforcement intelligent agent. An alarm treatment simulation environment is constructed through sparse unlabeled data, and an initial reinforcement learning intelligent agent and the alarm treatment simulation environment are interacted to simulate target alarm treatment schemes corresponding to different alarm states respectively, so that the initial reinforcement learning intelligent agent and the initial preference model are optimized. By using the method, the generalization capability of the fault alarm treatment scheme generation model can be improved.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a training method, apparatus, equipment and storage medium for a fault alarm handling scheme generation model. Background Technology

[0002] Alarm handling is a core component for ensuring stable system operation, and its efficiency and accuracy directly determine the system's fault recovery speed and operational reliability. With the rise of machine learning and deep learning technologies, intelligent alarm handling technologies are gradually replacing traditional manual methods. One example of intelligent alarm handling technology is preference-based alarm handling recommendation methods. However, these methods require a large amount of historical data labeled with preference relationships. In actual operation and maintenance scenarios, historical alarm and fault handling data is mostly unlabeled (only recording alarms and handling actions, without labeling the effectiveness of the handling). Furthermore, labeling labeled data is costly and time-consuming. Consequently, models cannot be effectively trained in sparse, unlabeled data scenarios, making it difficult to adapt to the needs of actual operation and maintenance scenarios and resulting in weak generalization ability.

[0003] Therefore, improving the generalization ability of the fault alarm handling solution generation model has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a training method, apparatus, device, and storage medium for a fault alarm handling scheme generation model, which can improve the generalization ability of the fault alarm handling scheme generation model.

[0005] In a first aspect, embodiments of this application provide a training method for a fault alarm handling scheme generation model, the method comprising:

[0006] Based on multiple preference sample pairs related to power system operation and maintenance, the initial preference model is trained to obtain a pre-trained preference model; each preference sample pair is constructed based on different historical handling schemes and different effect labels for the same alarm state.

[0007] The model parameters of the pre-trained preference model are transferred to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent;

[0008] An alarm handling simulation environment is constructed based on unlabeled operation and maintenance data related to power system operation and maintenance. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, the target alarm handling schemes corresponding to different alarm states are simulated. The reward value corresponding to each target alarm handling scheme is obtained based on a reward function that combines immediate and long-term rewards. The target alarm handling schemes corresponding to each alarm state are selected from multiple candidate handling schemes corresponding to the alarm state.

[0009] Based on the first target alarm handling scheme with high reward value and the second target alarm handling scheme with low reward value corresponding to the same alarm state, a new preference sample pair is constructed;

[0010] Update multiple preference sample pairs based on the new preference sample pairs, and retrain the initial preference model based on the updated multiple preference sample pairs to obtain a new pre-trained preference model;

[0011] The new pre-trained preference model is used as the pre-trained preference model. The model parameters of the pre-trained preference model are transferred to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent. The training continues until the stop training condition is met. The latest trained preference model is used as the generation model for fault alarm handling scheme of power system operation and maintenance.

[0012] In one embodiment, each set of preference sample pairs is constructed as follows: Labeled historical data associated with multiple historical fault alarm handling procedures are acquired; for each set of labeled historical data, data features corresponding to the labeled historical data are determined; and, based on a pre-built operation and maintenance knowledge graph, association features associated with the historical data are determined; and based on the data features and association features, alarm feature vectors corresponding to the labeled historical data are determined; based on the alarm feature vectors corresponding to each set of labeled historical data, the historical fault alarm data included in the multiple sets of labeled historical data are divided into multiple alarm categories; for each alarm category, based on multiple historical handling data corresponding to the alarm category and the operation and maintenance knowledge graph, a preset candidate handling scheme is generated. The rules generate N candidate handling schemes corresponding to alarm categories. Based on multiple historical handling data corresponding to alarm categories, invalid schemes among the N candidate handling schemes are eliminated to obtain M candidate handling schemes; M and N are both integers greater than 1, and M is less than or equal to N. From the M candidate handling schemes, the first candidate handling scheme and the second candidate handling scheme are determined. Based on the alarm feature vector corresponding to the alarm category, the first candidate handling scheme and the second candidate handling scheme, and the actual preference label, preference sample pairs corresponding to the alarm category are constructed. The actual preference label is determined based on the effect labels of the first candidate handling scheme and the second candidate handling scheme respectively. Based on the preference sample pairs corresponding to multiple alarm categories, multiple sets of preference sample pairs are obtained.

[0013] In one embodiment, the unlabeled operation and maintenance data includes an unlabeled fault dataset. Through the interaction between an initial reinforcement learning agent and an alarm handling simulation environment, target alarm handling schemes corresponding to different alarm states are simulated. Based on a reward function combining immediate and long-term rewards, reward values ​​corresponding to each target alarm handling scheme are obtained. This includes: selecting any fault scenario from multiple fault scenarios corresponding to the unlabeled fault dataset and generating an alarm state based on the selected fault scenario; selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, simulating the target alarm handling scheme based on a pre-built alarm handling simulation environment, and obtaining the reward value and a new alarm state corresponding to the target alarm handling scheme based on a reward function combining immediate and long-term rewards; generating experience data based on the alarm state, the target alarm handling scheme corresponding to the alarm state, the reward value, and the new alarm state; using the new alarm state as the alarm state and executing the step of selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, until the accumulated experience data reaches a preset quantity, obtaining the reward values ​​corresponding to the target alarm handling schemes for each of the multiple alarm states.

[0014] In one embodiment, the initial reinforcement learning agent includes a policy network; selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state includes: inputting the alarm state into the policy network to obtain the selection probabilities corresponding to the multiple candidate handling schemes corresponding to the alarm state; and selecting the target alarm handling scheme from the multiple candidate handling schemes based on the selection probabilities corresponding to the multiple candidate handling schemes.

[0015] In one embodiment, the method further includes: updating the network parameters of the initial reinforcement learning agent when the accumulated experience data reaches a preset amount, so that the target alarm handling scheme corresponding to the alarm state selected by the updated reinforcement learning agent can obtain a high cumulative reward.

[0016] In one embodiment, the method further includes: acquiring handling solution query data associated with the fault alarm; calling a large language model to parse the handling solution query data and extracting data features corresponding to the handling solution query data from the parsing results; inputting the data features into a fault alarm handling solution generation model to obtain a target handling solution and acquiring applicable instructions corresponding to the target handling solution; generating a fault alarm handling solution corresponding to the handling solution query data based on the target handling solution and the applicable instructions, according to the natural language generation rules of the large language model; and outputting the fault alarm handling solution.

[0017] In one embodiment, the method further includes: obtaining the actual handling plan and handling effect corresponding to the handling plan query data; determining new data based on the handling plan query data, fault alarm handling plan, actual handling plan and handling effect, and constructing new preference sample pairs based on the new data; adjusting the model parameters of the fault alarm handling plan generation model based on the new preference sample pairs to obtain the adjusted fault alarm handling plan generation model; and adjusting the natural language generation rules of the large language model based on the feedback information for the fault alarm handling plan to obtain the adjusted large language model.

[0018] Secondly, this application provides a training device for a fault alarm handling scheme generation model, the device comprising:

[0019] The training module is used to train the initial preference model based on multiple sets of preference sample pairs related to power system operation and maintenance, so as to obtain a pre-trained preference model; wherein, each set of preference sample pairs is constructed based on different historical handling schemes and different effect labels for the same alarm state;

[0020] The transfer module is used to transfer the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent.

[0021] The processing module is used to construct an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, it simulates the target alarm handling schemes corresponding to different alarm states. The reward value corresponding to each target alarm handling scheme is obtained based on a reward function that combines immediate and long-term rewards. The target alarm handling schemes corresponding to each alarm state are selected from a variety of candidate handling schemes corresponding to the alarm state.

[0022] The module is used to construct new preference sample pairs based on the first target alarm handling scheme with a high reward value and the second target alarm handling scheme with a low reward value corresponding to the same alarm state.

[0023] The training module is also used to update multiple preference sample pairs based on new preference sample pairs, and retrain the initial preference model based on the updated multiple preference sample pairs to obtain a new pre-trained preference model. The new pre-trained preference model is used as the pre-trained preference model to transfer the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent, until the training stop condition is met. The latest trained preference model is used as the generation model for fault alarm handling scheme of power system operation and maintenance.

[0024] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the method provided in the first aspect.

[0025] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the method provided in the first aspect.

[0027] The training method, apparatus, equipment, and storage medium for the aforementioned fault alarm handling scheme generation model train an initial preference model using historical labeled data to obtain a pre-trained preference model. This pre-trained preference model is then transferred to a pre-constructed reinforcement learning agent to obtain an initial reinforcement agent. An alarm handling simulation environment is constructed using sparse unlabeled data. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, target alarm handling schemes corresponding to different alarm states are simulated to optimize the initial reinforcement learning agent and the initial preference model. In this way, effective training and optimization of the preference model for generating fault alarm handling schemes can be achieved without a large amount of historical data labeled with preference relationships, resulting in a fault alarm handling scheme generation model. This improves the generalization ability of the fault alarm handling scheme generation model and adapts it to the needs of actual operation and maintenance scenarios. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a training method for a fault alarm handling scheme generation model provided in an embodiment of this application.

[0030] Figure 2 This is a flowchart illustrating a training method for another fault alarm handling scheme generation model provided in this application embodiment;

[0031] Figure 3 This is a schematic diagram of the structure of a training device for generating a fault alarm handling scheme model provided in an embodiment of this application;

[0032] Figure 4This is a schematic diagram of the structure of a training device for another fault alarm handling scheme generation model provided in this application embodiment;

[0033] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] Please see Figure 1 , Figure 1 This is a flowchart illustrating a training method for a fault alarm handling scheme generation model provided in an embodiment of this application. This method can be executed by a computer device. Optionally, the computer device can be a terminal device or a server. The terminal devices mentioned herein can include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc., and portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server mentioned herein can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, etc., without limitation. Figure 1 As shown, the training method for the fault alarm handling scheme generation model may include, but is not limited to, the following steps:

[0036] S101. Based on multiple sets of preference sample pairs related to power system operation and maintenance, the initial preference model is trained to obtain a pre-trained preference model; wherein, each set of preference sample pairs is constructed based on different historical handling schemes and different effect labels for the same alarm state.

[0037] Each pair of preference samples consists of an alarm feature vector corresponding to the alarm status, a feature vector of the candidate handling scheme, and an actual preference label determined based on the historical handling effects that can be associated with the candidate handling scheme.

[0038] Optionally, the feature vector of a candidate disposal solution can be composed of the step features, applicable condition features, and associated alarm features of the candidate disposal solution.

[0039] The initial preference model can be derived by using a large model of 32 bytes or more as a base and fine-tuning it by introducing an attention mechanism. The initial preference model optimizes the learning effect of preference relationships by making full use of the semantic understanding and feature extraction capabilities of the large model.

[0040] In some embodiments, the initial preference model can be a model that retains the original semantic encoding layer of the large model and adds a feature attention fine-tuning layer and a preference output layer. For example, the model structure of the initial preference model may include, but is not limited to:

[0041] (1) Input layer: Input preference sample pairs.

[0042] (2) Base layer: The large model is used as the basic base. It is responsible for deep semantic encoding of alarm features and candidate disposal scheme features, and mining potential correlations in the data. There is no need to build an additional basic feature extraction network. Relying on the powerful pre-training capability of the large model, the accuracy and comprehensiveness of feature encoding are improved.

[0043] (3) Feature Attention Fine-tuning Layer: After the output layer of the large model base, an attention fine-tuning layer is added. Through attention weight calculation, different weights are assigned to different features (e.g., the weight of alarm level is higher than the weight of handling step complexity), highlighting the influence of key features on preference relationships and adapting to the core needs of alarm handling scenarios. The formula for calculating attention weights is as follows:

[0044]

[0045] Among them, w i S represents the attention weight for the i-th feature, Si is the importance score of the i-th feature (obtained from historical data training), and n is the total number of features; j The importance score for the j-th feature.

[0046] (4) Hidden layer: The ReLU activation function is used to perform non-linear transformation on the attention-weighted feature vector. The number of hidden layer nodes is 256 to achieve deep feature fusion.

[0047] (5) Preference output layer: Based on the features encoded by the large model and weighted by attention, a preference output layer is constructed. The Sigmoid activation function is used to output the preference probability between the two candidate disposal schemes (i.e. the probability that scheme A is better than scheme B). The output range is [0,1]. When the output probability is greater than 0.5, scheme A is judged to be better than scheme B; otherwise, scheme B is judged to be better than scheme A. At the same time, the contextual understanding ability of the large model is used to optimize the calculation accuracy of the preference probability and reduce the bias of preference judgment caused by feature loss.

[0048] In some embodiments, the computer device trains an initial preference model based on multiple sets of preference sample pairs related to power system operation and maintenance to obtain a pre-trained preference model. This can be achieved by: inputting multiple sets of preference sample pairs related to power system operation and maintenance into the initial preference model to obtain the predicted preference label corresponding to the alarm feature vector included in each preference sample pair; determining the loss value between the predicted preference label and the actual preference label corresponding to the alarm feature vector included in each preference sample pair based on a preset loss function; and training the initial preference model in the direction of reducing the loss value corresponding to each preference sample pair to obtain a pre-trained preference model.

[0049] The loss function can be constructed using a Bradley-Terry model employed by the computer device. The loss function measures the deviation between the predicted preference labels and the actual preference labels, aiming to maximize the probability of predicting the correct preference label. Optionally, the loss function can be expressed as follows:

[0050]

[0051] in, This indicates that the model parameters of the preference model are... In the case of (X,A,B,y), the loss value between the predicted preference label and the actual preference label is calculated; (X,A,B,y) represents the preference sample pair, where X represents the alarm feature vector, A represents the feature vector of the first candidate solution, B represents the feature vector of the second candidate solution, and y represents the actual preference label (y=1 indicates that the first candidate solution is better than the second candidate solution, and y=0 indicates that the second candidate solution is better than the first candidate solution); D represents the preference sample pair dataset. This represents the model's preference score for the first candidate solution; This represents the model's preference score for the second candidate solution; This refers to the Sigmoid function.

[0052] Optionally, the computer device trains the initial preference model in the direction of reducing the loss value corresponding to each pair of preference samples to obtain a pre-trained preference model. This can include adjusting the model parameters of the larger models included in the base layer of the initial preference model with the goal of minimizing the loss function, thereby obtaining the pre-trained preference model. When adjusting the model parameters of the larger models included in the base layer of the initial preference model, the computer device can employ a lightweight fine-tuning method using Low-Rank Adaptation (LoRA).

[0053] Optionally, during the training of the initial preference model on the computer device, the learning rate can be set to 0.0001-0.001 (dynamically adjusted), the number of iterations can be 50-100 rounds, and the convergence condition (or stopping training condition) can be that the loss function value is less than 0.05.

[0054] In some embodiments, after obtaining a pre-trained preference model, the computer device may also acquire multiple sets of validation preference sample pairs and determine the preference prediction accuracy of the pre-trained preference model based on the multiple sets of validation preference sample pairs; if the preference prediction accuracy is greater than or equal to a preset accuracy threshold, the training of the initial preference model is completed; otherwise, the model parameters (such as attention weights, number of hidden layer nodes, etc.) are adjusted and retrained.

[0055] S102. Transfer the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent.

[0056] In some embodiments, the initial reinforcement learning agent includes a policy network and a value network.

[0057] Optionally, the policy network can be based on a large model of 32 bytes or more, reusing the model parameters of the large model, including the base layer of the pre-trained preference model. This reduces parameter redundancy and training costs.

[0058] The policy network takes as input the environmental state S (alarm features, system operating status, etc.) of the alarm handling simulation environment constructed based on unlabeled operation and maintenance data related to power system operation and maintenance. It outputs the selection probability of each action (candidate handling scheme) by relying on the semantic understanding and scenario adaptation capabilities of the large model.

[0059] The value network takes environmental state S as input and outputs the value estimate of the current state (i.e., the expected reward in the current state) to evaluate the rationality of action selection and assist in policy network optimization.

[0060] S103. Construct an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance, and simulate target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment. Obtain the reward value corresponding to each target alarm handling scheme based on a reward function that combines immediate rewards and long-term rewards.

[0061] The target alarm handling scheme corresponding to each alarm status is selected from a variety of candidate handling schemes corresponding to the alarm status.

[0062] In some embodiments, a computer device constructs an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance. This may include: constructing an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance and a power system operation and maintenance knowledge graph. The core elements of the alarm handling simulation environment may include environmental state, actions, and rewards.

[0063] The environmental state (which can be denoted as S) can be composed of the current alarm feature vector, the system operating state vector (a normal state baseline built based on non-fault data, such as equipment operating parameters and load conditions), and the fault impact range vector. The state space dimension is consistent with the preprocessed feature vector dimension.

[0064] Action (can be denoted as A): The action set is all valid candidate handling solutions for the current alarm category. Each action corresponds to a complete handling process. The action selection of the agent is to select a solution from the candidate solutions to execute.

[0065] Reward (denoted as R): The reward function is used to evaluate the effectiveness of the action (disposal plan). The reward function can be constructed based on a combination of immediate and long-term rewards. The reward function can be represented by the following formula:

[0066]

[0067] Where α is the weighting coefficient (0.6≤α≤0.8, which can be dynamically adjusted), R immediate For the immediate reward (calculated based on the deviation between the system state after simulation and the normal state; the smaller the deviation, the higher the immediate reward, ranging from [0,10]), R long The reward is for long-term purposes (calculated based on whether the system triggers an alarm again within a certain period after the incident and whether the fault is completely resolved. If no alarm is triggered again, the long-term reward is 10; otherwise, it is a negative reward, ranging from [-10, 10]).

[0068] Environment interaction logic: The simulation environment simulates the actual alarm handling process. After the agent selects an action (handling plan), the environment updates the system status, calculates the reward value, and feeds it back to the agent based on the action execution result. At the same time, it records the action execution process and result for subsequent model optimization.

[0069] S104. Based on the first target alarm handling scheme with high reward value and the second target alarm handling scheme with low reward value corresponding to the same alarm state, construct a new preference sample pair.

[0070] The first target alarm handling solution can also be called a high-quality alarm handling solution; the second target alarm handling solution can also be called a low-quality alarm handling solution.

[0071] S105. Update multiple preference sample pairs based on the new preference sample pairs, and retrain the initial preference model based on the updated multiple preference sample pairs to obtain a new pre-trained preference model.

[0072] In some embodiments, the computer device updates multiple sets of preference sample pairs based on new preference sample pairs, which may be done by adding new preference sample pairs to multiple sets of preference sample pairs to obtain updated multiple sets of preference sample pairs.

[0073] S106. Using the new pre-trained preference model as the pre-trained preference model, perform the step of transferring the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent, until the training stop condition is met, and use the latest trained preference model as the generation model of the fault alarm handling scheme for power system operation and maintenance.

[0074] In one optional implementation, the training termination condition may include any of the following: (1) the number of iterations reaches a preset number of iterations; (2) the cumulative reward fluctuation of the reinforcement learning agent during K rounds of training is less than a preset fluctuation threshold, where K is an integer greater than 1. Optionally, the preset fluctuation threshold may be determined based on expert experience or based on multiple trials, etc., and is not limited here.

[0075] In this embodiment, an initial preference model is trained using historical labeled data to obtain a pre-trained preference model. This pre-trained preference model is then transferred to a pre-built reinforcement learning agent to obtain an initial reinforcement agent. An alarm handling simulation environment is constructed using sparse unlabeled data. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, target alarm handling schemes corresponding to different alarm states are simulated to optimize the initial reinforcement learning agent and the initial preference model. In this way, without the need for a large amount of historical data labeled with preference relationships, the preference model used to generate fault alarm handling schemes can be effectively trained and optimized, resulting in a fault alarm handling scheme generation model. This improves the generalization ability of the fault alarm handling scheme generation model and adapts it to the needs of actual operation and maintenance scenarios.

[0076] In one alternative implementation, Figure 1In step S101 of the training method for the fault alarm handling scheme generation model shown, each set of preference sample pairs can be constructed by computer equipment in the following way: acquiring labeled historical data associated with multiple historical fault alarm handling respectively; for each set of labeled historical data, determining the data features corresponding to the labeled historical data, and, based on a pre-built operation and maintenance knowledge graph, determining the association features associated with the historical data, and based on the data features and association features, determining the alarm feature vector corresponding to the labeled historical data; based on the alarm feature vector corresponding to each set of labeled historical data, dividing the historical fault alarm data included in the multiple sets of labeled historical data into multiple alarm categories; for each alarm category, based on multiple historical handling data corresponding to the alarm category and the operation and maintenance knowledge graph... According to the preset candidate handling scheme generation rules, N candidate handling schemes are generated for each alarm category. Based on multiple historical handling data corresponding to each alarm category, invalid schemes are eliminated from the N candidate handling schemes, resulting in M ​​candidate handling schemes. M and N are both integers greater than 1, and M is less than or equal to N. From the M candidate handling schemes, the first candidate handling scheme and the second candidate handling scheme are determined. Based on the alarm feature vector corresponding to the alarm category, the first candidate handling scheme and the second candidate handling scheme, and the actual preference label, preference sample pairs corresponding to the alarm category are constructed. The actual preference label is determined based on the effect labels of the first candidate handling scheme and the second candidate handling scheme, respectively. Based on the preference sample pairs corresponding to multiple alarm categories, multiple sets of preference sample pairs are obtained.

[0077] The tagged historical data may include, but is not limited to, historical fault alarm data, historical fault data, and historical handling data. Optionally, historical fault alarm data may include, but is not limited to, alarm identifier (ID), alarm type, alarm level, alarm trigger time, associated device ID, alarm description, alarm parameters (such as monitoring indicators such as voltage and temperature); historical fault data may include, but is not limited to, fault ID, fault type, fault cause, fault impact range, fault duration; historical handling data may include, but is not limited to, handling ID, handling action, handling steps, handling time, handling personnel, handling effect (labeled as "excellent", "good", "poor" as preference tags), etc.

[0078] In some embodiments, the computer device determines the data features corresponding to the tagged historical data, which may include: performing invalid data removal and missing value filling on the tagged historical data to obtain processed tagged historical data; performing data standardization on the processed tagged historical data to obtain standardized tagged historical data; and performing feature extraction on the standardized tagged historical data to obtain the data features corresponding to the tagged historical data.

[0079] Invalid data includes, for example, data with missing alarm parameters, data with ambiguous fault types, data with incorrect labeling of handling effects, and duplicate data (such as the same alarm data that is repeatedly triggered).

[0080] Optionally, the computer equipment may perform missing value filling on the tagged historical data, which may include: for data with missing alarm parameters, filling the missing alarm parameter values ​​by using mean filling or interpolation filling; for data with missing fault causes, filling the missing fault cause values ​​by associating similar alarms, etc.

[0081] Optionally, the computer device performs data standardization processing on the processed labeled historical data to obtain standardized labeled historical data, which may include the following: obtaining standardized labeled historical data based on at least one of the first type of standardized labeled historical data, the second type of standardized labeled historical data, and the third type of standardized labeled historical data; wherein, the first type of standardized labeled historical data is obtained by encoding the classification data included in the processed labeled historical data; the second type of standardized labeled historical data is obtained by format conversion processing of the time data in the processed labeled historical data; and the third type of standardized labeled historical data is obtained by word segmentation and stop word removal processing of the text data in the processed labeled historical data.

[0082] For example, the computer device can encode the categorization data included in the processed tagged historical data, such as alarm level (e.g., urgent, important, general) and handling effect (excellent, good, poor), etc. (e.g., urgent=3, important=2, general=1; excellent=3, good=2, poor=1); convert the time data included in the processed tagged historical data, such as alarm trigger time, handling time, etc., into timestamp format; and perform word segmentation and stop word removal on the text data included in the processed tagged historical data, such as alarm description, fault cause, etc.

[0083] Optionally, the standardized labeled historical data includes text data and numerical data. The computer device performs feature extraction on the standardized labeled historical data to obtain the data features corresponding to the labeled historical data. This may include: using text feature extraction algorithms (such as Term Frequency-Inverse Document Frequency (TF-IDF), Word to Vector (Word2Vec), Bidirectional Encoder Representations from Transformers (BERT)) to determine the text features of the text data; using statistical methods to determine the numerical features of the numerical data; and determining the data features corresponding to the labeled historical data based on the text features and numerical features.

[0084] Text data includes, for example, alarm descriptions and fault causes; numerical data includes, for example, alarm parameters.

[0085] In some embodiments, the computer device determines association features associated with historical data based on a pre-built operation and maintenance knowledge graph, which may include: determining device association features and alarm and fault association features associated with historical data based on the pre-built operation and maintenance knowledge graph.

[0086] The operations and maintenance (O&M) knowledge graph may include, but is not limited to, device relationships, rules for associating faults and alarms, and procedures for handling such incidents. The O&M knowledge graph can be imported from an existing O&M knowledge base or supplemented and improved using an automated annotation engine.

[0087] In some embodiments, the computer device determines the alarm feature vector corresponding to the tagged historical data based on data features and association features, which may include: performing feature fusion processing on the data features and association features to obtain the alarm feature vector corresponding to the tagged data.

[0088] The dimension of the alarm feature vector can range from 128 to 512.

[0089] In some embodiments, the computer device divides historical fault alarm data included in multiple sets of labeled historical data into multiple alarm categories based on the alarm feature vector corresponding to each set of labeled historical data. This may include: using a clustering algorithm (such as K-means algorithm, hierarchical clustering algorithm, etc.) to divide the historical fault alarm data included in multiple sets of labeled historical data into multiple alarm categories based on the alarm feature vector corresponding to each set of labeled historical data. The multiple alarm categories may include voltage over-limit alarms, equipment overheating alarms, communication interruption alarms, etc.

[0090] In some embodiments, the preset candidate handling scheme generation rules may include, but are not limited to, candidate handling scheme generation rules based on the priority of handling actions (such as emergency alarms taking precedence over shutdown, troubleshooting, etc.), candidate handling scheme generation rules based on the correlation of handling steps (such as checking equipment status first and then performing repair actions), and candidate handling scheme generation rules based on the matching relationship between handling actions and alarm parameters (such as temperature exceeding threshold alarms corresponding to actions such as "reducing load" and "checking the heat dissipation system").

[0091] Each of the N candidate handling solutions includes complete handling steps, execution order, and applicable conditions. For example, assuming the alarm category corresponding to the alarm feature vector is an equipment overheating alarm, candidate handling solution 1 could be: check the heat dissipation system → clean the heat dissipation channels → reduce the equipment load; candidate handling solution 2 could be: shut down for cooling → check the cooling fan → restart the equipment; candidate handling solution 3 could be: adjust the equipment operating environment temperature → monitor temperature changes in real time.

[0092] In some embodiments, the computer device eliminates invalid solutions from N candidate solutions based on multiple historical handling data corresponding to alarm categories to obtain M candidate solutions. This may include: eliminating candidate solutions with poor handling effects from the N candidate solutions based on the handling effects in the multiple historical handling data corresponding to alarm categories to obtain M candidate solutions.

[0093] In some embodiments, the actual preference label may also be determined by the computer device in the following manner: a prompt word is constructed based on the alarm feature vector corresponding to the alarm category, the first feature vector corresponding to the first candidate handling scheme, and the second feature vector corresponding to the second candidate handling scheme; the prompt word is input into a large model to obtain the actual preference label. Here, the actual preference label is a preference label for the first candidate handling scheme and the second candidate handling scheme.

[0094] This implementation method uses labeled historical data as a basis to determine multiple alarm categories corresponding to the labeled historical data. For each alarm category, based on multiple historical handling data and operation and maintenance knowledge graphs corresponding to the alarm category, N candidate handling schemes are generated according to preset candidate handling scheme generation rules. Then, multiple sets of preference sample pairs are constructed based on the candidate handling schemes. In this way, the generated candidate handling schemes are highly targeted and have low redundancy, thereby providing a high-quality data foundation for subsequent model training.

[0095] In one alternative implementation, Figure 1In step S103 of the training method for the fault alarm handling scheme generation model shown, the unlabeled operation and maintenance data includes an unlabeled fault dataset; the computer device simulates the target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, and obtains the reward value corresponding to each target alarm handling scheme based on a reward function that combines immediate and long-term rewards. This may include: selecting any fault scenario from multiple fault scenarios corresponding to the unlabeled fault dataset, and generating an alarm state based on the selected fault scenario; selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state. An alarm handling scheme is developed, and based on a pre-built alarm handling simulation environment, a target alarm handling scheme is simulated. A reward function combining immediate and long-term rewards is used to obtain the reward value and new alarm status corresponding to the target alarm handling scheme. Based on the alarm status, the target alarm handling scheme corresponding to the alarm status, the reward value, and the new alarm status, empirical data is generated. The new alarm status is then used as the alarm status, and a step is executed to select the target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm status, until the empirical data accumulates to a preset quantity, obtaining the reward value corresponding to the target alarm handling scheme for each of the multiple alarm statuses.

[0096] In some embodiments, the multiple candidate handling schemes corresponding to the alarm status can be generated by the computer device based on the alarm feature vector corresponding to the alarm status, according to the preset candidate handling scheme generation rules.

[0097] In some embodiments, the initial reinforcement learning agent includes a policy network; the computer device selects a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, which may include: inputting the alarm state into the policy network to obtain the selection probabilities corresponding to the multiple candidate handling schemes corresponding to the alarm state; and selecting the target alarm handling scheme from the multiple candidate handling schemes based on the selection probabilities corresponding to the multiple candidate handling schemes.

[0098] Optionally, the computer device selects the target alarm handling scheme from multiple candidate handling schemes based on the selection probabilities corresponding to each of the multiple candidate handling schemes. This may include selecting the candidate handling scheme with the highest probability from the multiple candidate handling schemes as the target alarm handling scheme.

[0099] In some embodiments, the relevant description of the reward function based on the combination of immediate and long-term rewards can be found in the preceding description, and will not be repeated here.

[0100] In some embodiments, the preset quantity may be determined based on expert experience, multiple trials, or a human definition, etc., and is not limited here. For example, the preset quantity may be 1000.

[0101] In some embodiments, the computer device may also update the network parameters of the initial reinforcement learning agent when the accumulated experience data reaches a preset amount, so that the target alarm handling scheme corresponding to the alarm state selected by the updated reinforcement learning agent can obtain a high cumulative reward.

[0102] Optionally, the computer device may update the network parameters of the initial reinforcement learning agent by employing any one of the following algorithms: Proximal Policy Optimization (PPO), Deep Q-Network (DQN), or Asynchronous Advantage Actor-Critic (A3C).

[0103] By adopting this implementation method, the reward value corresponding to the target alarm handling scheme for multiple alarm states can be quickly determined through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment. In addition, through the long-term reward mechanism of reinforcement learning, the model can take into account both the effect of a single handling and the long-term system operation status, avoiding the situation where the single handling is optimal but the long-term impact is poor, and reducing the probability of subsequent alarm triggering in the system.

[0104] In one alternative implementation, Figure 1 The training method for the fault alarm handling solution generation model may further include: acquiring handling solution query data associated with the fault alarm; calling a large language model to parse the handling solution query data and extracting data features corresponding to the handling solution query data from the parsing results; inputting the data features into the fault alarm handling solution generation model to obtain the target handling solution and acquiring the applicable instructions corresponding to the target handling solution; generating the fault alarm handling solution corresponding to the handling solution query data based on the target handling solution and the applicable instructions, according to the natural language generation rules of the large language model; and outputting the fault alarm handling solution.

[0105] In some embodiments, the computer device acquires the handling solution query data associated with the fault alarm by receiving the handling solution query data associated with the fault alarm from the operation and maintenance terminal; wherein, the handling solution query data may be a natural language query entered by the operation and maintenance personnel through the operation and maintenance terminal.

[0106] For example, the data query for handling solutions could be something like: "How to handle equipment overheating alarms?" or "What is the optimal handling solution for voltage over-limit alarms?"

[0107] In some embodiments, the computer device may obtain applicable instruction information corresponding to the target treatment plan by: obtaining applicable instruction information corresponding to the target treatment plan based on an operation and maintenance knowledge graph.

[0108] In some embodiments, after the computer device outputs a fault alarm handling solution, if it detects problem data related to the fault alarm handling solution input by the target object, it calls a large language model to parse the problem data and extracts the data features corresponding to the problem data from the parsing results; the data features are then input into the fault alarm handling solution generation model to obtain supplementary response data for the target handling solution.

[0109] In some embodiments, the computer device may also acquire the actual handling scheme and handling effect corresponding to the handling scheme query data, determine new data based on the handling scheme query data, fault alarm handling scheme, actual handling scheme and handling effect, and construct new preference sample pairs based on the new data; adjust the model parameters of the fault alarm handling scheme generation model based on the new preference sample pairs to obtain the adjusted fault alarm handling scheme generation model; and adjust the natural language generation rules of the large language model based on the feedback information for the fault alarm handling scheme to obtain the adjusted large language model.

[0110] Optionally, when adjusting the model parameters of the fault alarm handling solution generation model based on the newly added preference sample pairs, the computer equipment can adopt the low-rank adaptation (LoRA) fine-tuning method.

[0111] Optional feedback information for fault alarm handling solutions could include phrases such as "inaccurate response" or "unreasonable handling solution".

[0112] Optionally, while adjusting the natural language generation rules of the large language model, the computer equipment can also fine-tune the model parameters (such as feature weights) of the fault alarm handling solution generation model to ensure that the large language model and the fault alarm handling solution generation model can adapt to the dynamic changes of the alarm handling scenario.

[0113] In some embodiments, the computer device can also acquire in real time indicators such as inference accuracy, question-and-answer response time, and feedback on handling effect of the fault alarm handling solution generation model; if the indicators are abnormal, the model parameters of the fault alarm handling solution generation model are updated to ensure that the performance of the fault alarm handling solution generation model tends to be stable.

[0114] Using this implementation method, maintenance personnel do not need to master complex model operations. They can quickly obtain accurate fault alarm handling solutions by inputting the handling solutions associated with the fault alarms and querying the data, thereby improving the user experience of maintenance personnel.

[0115] The training method for the fault alarm handling scheme generation model provided in this application embodiment is described below. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a training method for a fault alarm handling scheme generation model provided in an embodiment of this application, as shown below. Figure 2 As shown, the training method for the fault alarm handling scheme generation model may include, but is not limited to, the following steps:

[0116] S201. Obtain labeled historical data, unlabeled operation and maintenance data, and operation and maintenance knowledge graphs that are associated with the handling of multiple historical fault alarms.

[0117] The labeled historical data may include historical fault alarm data, historical fault data, and historical handling data. Historical fault alarm data may include, but is not limited to, alarm ID, alarm type, alarm level, alarm trigger time, associated device ID, alarm description, and alarm parameters (such as monitoring indicators like voltage and temperature). Historical fault data may include, but is not limited to, fault ID, fault type, fault cause, fault impact range, and fault duration. Historical handling data may include, but is not limited to, handling ID, handling action, handling steps, handling time, handling personnel, and handling effect (labeled as "Excellent," "Good," or "Poor," serving as preference tags). This labeled historical data is used as the initial parameters for subsequent training of the preference model.

[0118] The unlabeled maintenance data can include unlabeled fault data (which only records the fault ID, fault type, and fault impact range, without indicating the handling actions and handling effects) and non-fault data (which records monitoring data when the equipment is running normally and system status data during periods without alarms). The unlabeled maintenance data is used to subsequently build an alarm handling simulation environment to simulate the execution effects of different handling schemes.

[0119] Among them, the operation and maintenance knowledge graph (including device relationships, fault and alarm association rules, and handling action specifications) can be used to assist in the compilation of candidate handling solutions and the natural language conversion of question and answer models. This data can be imported from the existing operation and maintenance knowledge base or supplemented and improved by an automated annotation engine.

[0120] S202. Perform data cleaning, standardization, and feature extraction on labeled historical data, unlabeled operation and maintenance data, and data in the operation and maintenance knowledge graph to obtain data features for each data set.

[0121] (1) Data cleaning: Remove invalid data (such as data with missing alarm parameters, ambiguous fault types, and incorrect labeling of handling effects), process missing values ​​(fill in missing alarm parameters by means of mean filling, interpolation filling, etc., and fill in missing fault causes by means of similar alarm association filling), and remove duplicate data (such as the same alarm data that is repeatedly triggered).

[0122] (2) Data standardization: Encode the alarm level (such as emergency, important, general) and handling effect (excellent, good, poor) data (such as emergency=3, important=2, general=1; excellent=3, good=2, poor=1), convert the alarm trigger time, handling time and other time data into timestamp format, and perform word segmentation and stop word removal on the text data such as alarm description and fault cause;

[0123] (3) Feature extraction: The TF-IDF algorithm is used to extract text features of alarm description and fault cause, and statistical methods are used to extract numerical features of alarm parameters (such as mean, variance and peak value). Combined with knowledge graph, the device association features and alarm and fault association features are extracted. Finally, a unified feature vector (128-512 dimensions, dynamically adjusted according to the data scale) is constructed for subsequent model training and candidate scheme compilation.

[0124] After determining the data characteristics of multiple data sets, the computer equipment can classify and store the data characteristics of the multiple data sets according to "labeled historical data", "unlabeled fault data", "non-fault data" and "knowledge graph data".

[0125] S203. Based on the data characteristics and operation and maintenance knowledge graph corresponding to the labeled historical data, determine multiple candidate handling solutions corresponding to different alarm types.

[0126] In some embodiments, step S203 may include, but is not limited to, the following steps:

[0127] (1) Alarm classification and clustering: The K-means clustering algorithm is adopted. Based on the alarm feature vector (such as alarm type, alarm level, associated equipment, text features), the historical alarm data is divided into multiple categories (such as voltage over-limit alarm, equipment overheating alarm, communication interruption alarm, etc. in the power system) to ensure that the handling logic of the same type of alarm is similar.

[0128] (2) Construction of candidate solution generation rules: Combining the operation and maintenance knowledge graph and historical handling data, candidate handling solution generation rules are constructed for each alarm category. The rules include: the priority of handling actions (e.g., emergency alarms should prioritize shutdown and troubleshooting actions), the correlation of handling steps (e.g., check the equipment status first and then perform repair actions), and the matching relationship between handling actions and alarm parameters (e.g., temperature over threshold alarms correspond to actions such as "reduce load" and "check the heat dissipation system").

[0129] (3) Candidate handling plan preparation: For each alarm category, based on the generation rules, prepare 3-5 candidate handling plans (the number can be dynamically adjusted). Each candidate plan includes complete handling steps, execution order, and applicable conditions.

[0130] (4) Candidate solution screening: The candidate solutions compiled are initially screened using the historical treatment effect data included in the labeled historical data. Invalid solutions (such as solutions with all historical treatment effects being "poor") are eliminated, and valid candidate solutions are retained to obtain multiple candidate treatment solutions for each alarm category.

[0131] S204. Based on multiple candidate handling schemes corresponding to each alarm category, construct multiple sets of preference sample pairs.

[0132] The preference sample pair can be represented as (X,A,B,y), where X represents the alarm feature vector corresponding to the alarm category, A represents the feature vector corresponding to the first candidate handling scheme determined from multiple candidate handling schemes corresponding to the alarm feature vector X, B represents the feature vector corresponding to the second candidate handling scheme determined from multiple candidate handling schemes corresponding to the alarm feature vector X, and y represents the actual preference label (y=1 indicates that the first candidate handling scheme is better than the second candidate handling scheme, and y=0 indicates that the second candidate handling scheme is better than the first candidate handling scheme).

[0133] S205. Based on multiple sets of preference sample pairs, train the initial preference model to obtain a pre-trained preference model.

[0134] In some embodiments, the relevant description of step S205 can be found in the relevant description of step S101 above, and will not be repeated here.

[0135] S206. Construct an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance. Simulate target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment. Obtain the reward value corresponding to each target alarm handling scheme based on a reward function that combines immediate and long-term rewards.

[0136] S207. Based on the first target alarm handling scheme with high reward value and the second target alarm handling scheme with low reward value corresponding to the same alarm state, construct a new preference sample pair.

[0137] S208. Update multiple preference sample pairs based on the new preference sample pairs, and retrain the initial preference model based on the updated multiple preference sample pairs to obtain a new pre-trained preference model.

[0138] S209. Using the new pre-trained preference model as the pre-trained preference model, perform the step of transferring the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent, until the training stop condition is met, and use the latest trained preference model as the generation model of the fault alarm handling scheme for power system operation and maintenance.

[0139] In some embodiments, the relevant descriptions of steps S206 to S209 can be found in the relevant descriptions above, and will not be repeated here.

[0140] S210. Based on the fault alarm handling solution generation model and the big language model, construct an alarm handling question and answer model.

[0141] The alarm handling question-and-answer model can adopt a two-stage structure of "feature mapping + natural language generation," using a 32-bit or larger large language model (Qwen3-32B or Gemini-32B) as its core foundation. Combined with an operations and maintenance domain knowledge graph, it achieves accurate matching between alarm queries and handling suggestions, fully leveraging the natural language understanding and generation capabilities of the large model to improve the fluency and accuracy of the question-and-answer interaction. Its specific structure is as follows:

[0142] (1) Query understanding layer: Receives natural language queries (such as "How to handle equipment overheating alarm?" "What is the optimal solution for voltage over-limit alarm?") input by maintenance personnel through the terminal. Relying on the semantic understanding capabilities of the Qwen3-32B or Gemini-32B large model, it directly performs word segmentation, semantic encoding and intent recognition on the query text. No additional BERT model is required. It accurately extracts key information such as alarm type and core requirements in the query and converts them into feature vectors consistent with the preprocessed ones, thereby improving the accuracy and efficiency of query understanding.

[0143] (2) Solution matching layer: Input the query feature vector into the optimized preference model (i.e. the fault alarm handling solution generation model obtained in step S209), obtain the preference ranking of the handling candidate solutions, and select the 1-2 handling solutions with the best ranking as candidate answers; at the same time, combine the knowledge graph to supplement the applicable conditions and precautions of the handling solution (such as "this solution is applicable to scenarios where the equipment load is less than 80%).

[0144] (3) Natural Language Generation Layer: Reusing the generation capabilities of the large language model, fine-tuning it based on the knowledge of the operation and maintenance domain, and converting candidate handling solutions (steps, applicable conditions) into natural language answers, ensuring that the answers are concise, accurate, easy to understand, and in line with the operating habits of operation and maintenance personnel. At the same time, relying on the contextual association capabilities of the large model, a coherent response to multiple rounds of questions and answers can be achieved. For example: "For equipment overheating alarms, the recommended optimal handling solution is: 1. Check whether the heat dissipation system is operating normally; 2. Clean the dust in the heat dissipation channel; 3. Reduce the equipment load and monitor temperature changes in real time. This solution is suitable for scenarios where the equipment load is below 80% and the ambient temperature is below 35℃."

[0145] S211. Obtain query data for handling solutions associated with fault alarms.

[0146] S212. Input the handling solution query data into the handling solution question and answer model to obtain the fault alarm handling solution corresponding to the handling solution query data, and output the fault alarm handling solution.

[0147] In some embodiments, the computer device may also acquire the actual handling plan and handling effect corresponding to the handling plan query data, determine new data based on the handling plan query data, fault alarm handling plan, actual handling plan and handling effect, and construct new preference sample pairs based on the new data; adjust the model parameters of the fault alarm handling plan generation model included in the handling plan question answering model based on the new preference sample pairs to obtain the adjusted fault alarm handling plan generation model; and adjust the natural language generation rules of the large language model included in the handling plan question answering model based on the feedback information for the fault alarm handling plan to obtain the adjusted large language model; and generate a new handling plan question answering model based on the adjusted fault alarm handling plan generation model and the adjusted large language model.

[0148] In this embodiment, an initial preference model is trained using historical labeled data to obtain a pre-trained preference model. This pre-trained preference model is then transferred to a pre-built reinforcement learning agent to obtain an initial reinforcement agent. An alarm handling simulation environment is constructed using sparse unlabeled data. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, target alarm handling schemes corresponding to different alarm states are simulated to optimize the initial reinforcement learning agent and the initial preference model. In this way, without the need for a large amount of historical data labeled with preference relationships, the preference model used to generate fault alarm handling schemes can be effectively trained and optimized, resulting in a fault alarm handling scheme generation model. This improves the generalization ability of the fault alarm handling scheme generation model and adapts it to the needs of actual operation and maintenance scenarios.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a training device for a fault alarm handling scheme generation model, which is used to implement the training method for the fault alarm handling scheme generation model described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more training device embodiments for fault alarm handling scheme generation models provided below can be found in the limitations of the training method for fault alarm handling scheme generation models described above, and will not be repeated here.

[0151] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a training device for generating a fault alarm handling scheme model provided in an embodiment of this application. Figure 3 As shown, the training device for the fault alarm handling scheme generation model may include, but is not limited to, a data acquisition and preprocessing module 301, a candidate handling scheme compilation module 302, a preference model module 303, a reinforcement learning simulation module 304, an alarm handling question and answer module 305, and a model optimization module 306.

[0152] The data acquisition and preprocessing module 301 is used to preprocess various types of data related to alarm handling, providing high-quality data support for subsequent modules.

[0153] The candidate handling scheme compilation module 302 is used to compile multiple targeted handling candidate schemes for different types of alarms based on preprocessed historical data and knowledge graph data.

[0154] Preference model module 303 is used to train a preference model based on historical labeled data and candidate disposal options, and to learn the preference relationship between the candidate disposal options.

[0155] The reinforcement learning simulation module 304 is used to construct an alarm handling simulation environment using sparse unlabeled fault and non-fault data. Through the interaction between the reinforcement learning agent and the environment, it simulates the execution effect of different handling candidate schemes and dynamically optimizes the preference model parameters.

[0156] The alarm handling Q&A module 305 is used to build an alarm handling Q&A model, enabling operation and maintenance personnel to interact with the model in natural language, and converting the model's output handling solutions into easy-to-understand natural language answers.

[0157] The model optimization module 306 is used to continuously optimize the model to adapt to the dynamic changes in alarm handling scenarios.

[0158] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a training device for another fault alarm handling scheme generation model provided in this application embodiment. For example... Figure 4 As shown, the training device for the fault alarm handling scheme generation model may include, but is not limited to:

[0159] Training module 401 is used to train the initial preference model based on multiple sets of preference sample pairs related to power system operation and maintenance, so as to obtain a pre-trained preference model; wherein, each set of preference sample pairs is constructed based on different historical handling schemes and different effect labels for the same alarm state;

[0160] The transfer module 402 is used to transfer the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent.

[0161] The processing module 403 is used to construct an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance, and simulate the target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment. The reward value corresponding to each target alarm handling scheme is obtained based on a reward function that combines immediate reward and long-term reward. The target alarm handling schemes corresponding to each alarm state are selected from a variety of candidate handling schemes corresponding to the alarm state.

[0162] Module 404 is used to construct new preference sample pairs based on the first target alarm handling scheme with high reward value and the second target alarm handling scheme with low reward value corresponding to the same alarm state;

[0163] The training module 401 is also used to update multiple preference sample pairs based on new preference sample pairs, and retrain the initial preference model based on the updated multiple preference sample pairs to obtain a new pre-trained preference model; using the new pre-trained preference model as the pre-trained preference model, the step of transferring the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent is executed to obtain the initial reinforcement learning agent, until the training stop condition is met, and the latest trained preference model is used as the generation model of the fault alarm handling scheme for power system operation and maintenance.

[0164] In one embodiment, the construction module 404 is further configured to: acquire tagged historical data associated with multiple historical fault alarm handling procedures; for each group of tagged historical data, determine the data features corresponding to the tagged historical data; and, based on a pre-built operation and maintenance knowledge graph, determine the association features associated with the historical data, and based on the data features and association features, determine the alarm feature vector corresponding to the tagged historical data; based on the alarm feature vector corresponding to each group of tagged historical data, divide the historical fault alarm data included in the multiple groups of tagged historical data into multiple alarm categories; and for each alarm category, based on multiple historical handling data corresponding to the alarm category and the operation and maintenance knowledge graph, generate a candidate handling scheme according to a preset rule. N candidate handling schemes are generated for each alarm category. Based on multiple historical handling data for each alarm category, invalid schemes are eliminated from the N candidate handling schemes to obtain M candidate handling schemes. M and N are both integers greater than 1, and M is less than or equal to N. From the M candidate handling schemes, a first candidate handling scheme and a second candidate handling scheme are determined. Based on the alarm feature vector corresponding to the alarm category, the first candidate handling scheme and the second candidate handling scheme, and the actual preference label, a preference sample pair corresponding to the alarm category is constructed. The actual preference label is determined based on the effect labels of the first candidate handling scheme and the second candidate handling scheme, respectively. Based on the preference sample pairs corresponding to multiple alarm categories, multiple sets of preference sample pairs are obtained.

[0165] In one embodiment, the unlabeled operation and maintenance data includes an unlabeled fault dataset. When the processing module 403 simulates target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, and obtains the reward value corresponding to each target alarm handling scheme based on a reward function combining immediate and long-term rewards, it specifically performs the following steps: selecting any fault scenario from multiple fault scenarios corresponding to the unlabeled fault dataset, and generating an alarm state based on the selected fault scenario; selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, simulating the target alarm handling scheme based on a pre-built alarm handling simulation environment, and obtaining the reward value and new alarm state corresponding to the target alarm handling scheme based on a reward function combining immediate and long-term rewards; generating experience data based on the alarm state, the target alarm handling scheme corresponding to the alarm state, the reward value, and the new alarm state; using the new alarm state as the alarm state, and executing the step of selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, until the accumulated experience data reaches a preset quantity, obtaining the reward value corresponding to the target alarm handling scheme for each of the multiple alarm states.

[0166] In one embodiment, the initial reinforcement learning agent includes a policy network; when the processing module 403 selects a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, it specifically performs the following steps: inputting the alarm state into the policy network to obtain the selection probabilities corresponding to the multiple candidate handling schemes corresponding to the alarm state; and selecting the target alarm handling scheme from the multiple candidate handling schemes based on the selection probabilities corresponding to the multiple candidate handling schemes.

[0167] In one embodiment, the processing module 403 is further configured to: update the network parameters of the initial reinforcement learning agent when the accumulated experience data reaches a preset amount, so that the target alarm handling scheme corresponding to the alarm state selected by the updated reinforcement learning agent can obtain a high cumulative reward.

[0168] In one embodiment, the processing module 403 is further configured to: acquire query data of handling solutions associated with fault alarms; call a large language model to parse the handling solution query data and extract data features corresponding to the handling solution query data from the parsing results; input the data features into a fault alarm handling solution generation model to obtain a target handling solution and acquire applicable instruction information corresponding to the target handling solution; based on the target handling solution and the applicable instruction information, generate a fault alarm handling solution corresponding to the handling solution query data according to the natural language generation rules of the large language model; and output the fault alarm handling solution.

[0169] In one embodiment, the processing module 403 is further configured to: obtain the actual handling scheme and handling effect corresponding to the handling scheme query data; determine new data based on the handling scheme query data, fault alarm handling scheme, actual handling scheme and handling effect, and construct new preference sample pairs based on the new data; adjust the model parameters of the fault alarm handling scheme generation model based on the new preference sample pairs to obtain the adjusted fault alarm handling scheme generation model; and adjust the natural language generation rules of the large language model based on the feedback information for the fault alarm handling scheme to obtain the adjusted large language model.

[0170] Each module in the training device for the aforementioned fault alarm handling scheme generation model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the terminal device in hardware form or independent of it, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a training method for generating a fault alarm handling scheme model. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0172] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one exemplary embodiment, this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the training method for generating the fault alarm handling scheme described above.

[0174] In one exemplary embodiment, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the training method for the above-described fault alarm handling scheme generation model.

[0175] In one exemplary embodiment, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the training method for generating the fault alarm handling scheme described above.

[0176] It should be noted that the data involved in this application (including but not limited to acquired data, data used for analysis, and stored data) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a fault alarm handling scheme generation model, characterized in that, The method includes: Based on multiple sets of preference sample pairs related to power system operation and maintenance, the initial preference model is trained to obtain a pre-trained preference model; wherein, each set of preference sample pairs is constructed based on different historical handling schemes and different effect labels for the same alarm state; The model parameters of the pre-trained preference model are transferred to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent; An alarm handling simulation environment is constructed based on unlabeled operation and maintenance data related to power system operation and maintenance. Through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment, the target alarm handling schemes corresponding to different alarm states are simulated. The reward value corresponding to each target alarm handling scheme is obtained based on a reward function that combines immediate and long-term rewards. The target alarm handling scheme corresponding to each alarm state is selected from multiple candidate handling schemes corresponding to the alarm state. Based on the first target alarm handling scheme with high reward value and the second target alarm handling scheme with low reward value corresponding to the same alarm state, a new preference sample pair is constructed; Based on the new preference sample pairs, update multiple sets of preference sample pairs, and based on the updated multiple sets of preference sample pairs, retrain the initial preference model to obtain a new pre-trained preference model; The new pre-trained preference model is used as the pre-trained preference model. The step of transferring the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent is performed to obtain the initial reinforcement learning agent until the training stop condition is met. The latest trained preference model is used as the generation model of the fault alarm handling scheme for power system operation and maintenance.

2. The method according to claim 1, characterized in that, Each set of preference sample pairs is constructed in the following manner: Acquire tagged historical data that are associated with the handling of multiple historical fault alarms; For each set of labeled historical data, the data features corresponding to the labeled historical data are determined, and based on the pre-built operation and maintenance knowledge graph, the association features associated with the historical data are determined, and based on the data features and the association features, the alarm feature vector corresponding to the labeled historical data is determined. Based on the alarm feature vector corresponding to each group of labeled historical data, the historical fault alarm data included in the multiple groups of labeled historical data are divided into multiple alarm categories. For each alarm category, based on multiple historical handling data corresponding to the alarm category and the operation and maintenance knowledge graph, N candidate handling schemes corresponding to the alarm category are generated according to the preset candidate handling scheme generation rules. Based on multiple historical handling data corresponding to the alarm category, invalid schemes among the N candidate handling schemes are eliminated to obtain M candidate handling schemes. M and N are both integers greater than 1, and M is less than or equal to N; From the M candidate disposal schemes, determine the first candidate disposal scheme and the second candidate disposal scheme; Based on the alarm feature vector corresponding to the alarm category, the first candidate handling scheme and the second candidate handling scheme, and the actual preference label, the preference sample pair corresponding to the alarm category is constructed; The actual preference label is determined based on the effect labels of the first candidate treatment plan and the second candidate treatment plan, respectively. Based on the preference sample pairs corresponding to the multiple alarm categories, multiple sets of preference sample pairs are obtained.

3. The method according to claim 1, characterized in that, The unlabeled operation and maintenance data includes an unlabeled fault dataset; the interaction between the initial reinforcement learning agent and the alarm handling simulation environment simulates target alarm handling schemes corresponding to different alarm states, and the reward value corresponding to each target alarm handling scheme is obtained based on a reward function combining immediate and long-term rewards, including: Select any one of the multiple fault scenarios corresponding to the unlabeled fault dataset, and generate an alarm status based on the selected fault scenario; Select a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state, simulate the target alarm handling scheme based on a pre-built alarm handling simulation environment, and obtain the reward value and new alarm state corresponding to the target alarm handling scheme based on a reward function that combines immediate rewards and long-term rewards. Based on the alarm status, the target alarm handling plan corresponding to the alarm status, the reward value, and the new alarm status, experience data is generated; The new alarm state is taken as the alarm state, and the step of selecting a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state is executed until the accumulated experience data reaches a preset amount, so as to obtain the reward value corresponding to the target alarm handling scheme for each of the multiple alarm states.

4. The method according to claim 3, characterized in that, The initial reinforcement learning agent includes a policy network; the selection of a target alarm handling scheme from multiple candidate handling schemes corresponding to the alarm state includes: The alarm status is input into the policy network to obtain the selection probabilities of various candidate handling schemes corresponding to the alarm status. Based on the selection probabilities corresponding to the various candidate handling schemes, a target alarm handling scheme is selected from the various candidate handling schemes.

5. The method according to claim 3, characterized in that, The method further includes: When the accumulated experience data reaches the preset amount, the network parameters of the initial reinforcement learning agent are updated so that the target alarm handling scheme corresponding to the alarm state selected by the updated reinforcement learning agent can obtain a high cumulative reward.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Retrieve query data for handling solutions associated with fault alarms; The large language model is invoked to parse the query data of the treatment plan, and the data features corresponding to the query data of the treatment plan are extracted from the parsing results; The data features are input into the fault alarm handling solution generation model to obtain the target handling solution, and the applicable instructions corresponding to the target handling solution are obtained. Based on the target handling plan and the applicable instructions, and in accordance with the natural language generation rules of the large language model, a fault alarm handling plan corresponding to the handling plan query data is generated. Output the fault alarm handling plan.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the actual handling plan and handling effect corresponding to the handling plan query data; based on the handling plan query data, the fault alarm handling plan, the actual handling plan and the handling effect, determine the new data, and construct the new preference sample pair based on the new data; Based on the newly added preference sample pairs, the model parameters of the fault alarm handling scheme generation model are adjusted to obtain the adjusted fault alarm handling scheme generation model; and, Based on the feedback information regarding the fault alarm handling scheme, the natural language generation rules of the large language model are adjusted to obtain the adjusted large language model.

8. A training device for generating a fault alarm handling scheme model, characterized in that, The training module is used to train the initial preference model based on multiple sets of preference sample pairs related to power system operation and maintenance, so as to obtain a pre-trained preference model; wherein, each set of preference sample pairs is constructed based on different historical handling schemes and different effect labels for the same alarm state; The transfer module is used to transfer the model parameters of the pre-trained preference model to the pre-built reinforcement learning agent to obtain the initial reinforcement learning agent. The processing module is used to construct an alarm handling simulation environment based on unlabeled operation and maintenance data related to power system operation and maintenance, and simulate target alarm handling schemes corresponding to different alarm states through the interaction between the initial reinforcement learning agent and the alarm handling simulation environment. The reward value corresponding to each target alarm handling scheme is obtained based on a reward function that combines immediate and long-term rewards. The target alarm handling scheme corresponding to each alarm state is selected from a variety of candidate handling schemes corresponding to the alarm state. The module is used to construct new preference sample pairs based on the first target alarm handling scheme with a high reward value and the second target alarm handling scheme with a low reward value corresponding to the same alarm state. The training module is further configured to update multiple sets of preference sample pairs based on the new preference sample pairs, and retrain the initial preference model based on the updated multiple sets of preference sample pairs to obtain a new pre-trained preference model; use the new pre-trained preference model as the pre-trained preference model, and execute the step of transferring the model parameters of the pre-trained preference model to a pre-built reinforcement learning agent to obtain an initial reinforcement learning agent, until the training stop condition is met, and use the latest trained preference model as the generation model for the fault alarm handling scheme of power system operation and maintenance.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program; when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images