Multi-agent policy network training method, transformer fault early warning method and device

Abstract

This application provides a multi-agent policy network training method, a transformer fault early warning method, and a device. The training method first pre-trains each individual agent policy network and the overall agent policy network in parallel. Then, it establishes the relationship between the immediate reward of the agent policy control and the network loss of the overall agent policy network. Next, it updates the network with the goal of maximizing the immediate reward. The network update process is the process of adjusting the network parameters. The output actions of the individual agent policy networks are used as the inputs of the overall agent policy network. By maximizing the immediate reward, the corresponding optimal network parameters are determined, thereby completing the training of the agent policy network. The trained multi-agent policy network can comprehensively utilize information from multiple sensors to achieve comprehensive monitoring and fault early warning of the power transformer system.

Description

Technical Field

[0001] This invention pertains to transformer monitoring technology, and particularly relates to a multi-agent strategy network training method, a transformer fault early warning method, and a device. Background Technology

[0002] Power transformers are crucial power conversion devices in power systems, used to convert high-voltage electrical energy to low-voltage electrical energy and vice versa. The normal operation of transformers is vital to the stability and reliability of the power grid. However, transformers may encounter various faults and abnormal conditions during operation, such as overload, discharge, and high temperature, which can lead to equipment damage or even accidents. Therefore, timely fault warning and control are essential to ensuring the safe operation of the power system.

[0003] Traditional power transformer monitoring methods mainly rely on manual inspection and fixed sensor monitoring, which have the following limitations and disadvantages: (1) The periodicity of manual inspection limits the timely detection of faults, and the workload of manual inspection in large power systems is large and the efficiency is low; (2) Fixed sensors can only monitor data at specific locations and cannot fully perceive the state of the transformer; (3) The processing and decision-making of sensor data mainly rely on pre-set rules and thresholds, lacking flexibility and adaptability.

[0004] In recent years, deep reinforcement learning has been widely used in power systems. Deep reinforcement learning is a method based on artificial intelligence and machine learning, which can optimize decision-making strategies and achieve autonomous control through interactive learning between the agent and the system. Compared with traditional methods, it has the following advantages: (1) It can handle complex nonlinear systems and is suitable for multivariable and nonlinear problems of transformers in power systems; (2) Through interactive learning with the environment, it can autonomously explore and adapt to different working states and fault modes; (3) It has strong generalization ability and can learn general rules from limited data to adapt to the application of different systems.

[0005] However, when it comes to fault monitoring of power equipment, especially transformers, existing deep reinforcement learning methods mainly focus on the training and control of a single agent, neglecting the comprehensive utilization of information from multiple sensors in power transformer systems and the interaction between multiple agents. Summary of the Invention

[0006] Based on this, the present invention aims to propose a multi-agent policy network training method, a transformer fault early warning method and device, in which multiple agents perform parallel deep reinforcement learning to construct an early warning model that can comprehensively utilize information from multiple sensors, thereby realizing comprehensive early warning and risk control of power transformer systems.

[0007] In a first aspect, this invention proposes a multi-agent policy network training method. The agent policy network trained by this method is used for transformer fault early warning. The method includes:

[0008] Obtain datasets of different properties of the transformer collected by each sensor;

[0009] Based on the properties of the transformer, the policy networks of each single agent are trained in parallel using the dataset. Each single agent policy network corresponds one-to-one with each property of the transformer. The overall agent policy network is pre-trained based on the output of the pre-trained single agent policy networks.

[0010] Establish the relationship between immediate policy rewards and the network loss of the overall agent policy network;

[0011] With the goal of maximizing immediate policy rewards, and using the output of the single agent policy network as the input of the overall agent policy network, we collaboratively train the individual agent policy networks and the overall agent policy network.

[0012] The policy immediate reward is calculated based on the network loss of the overall agent policy network, and the network parameters of each individual agent policy network and the overall agent policy network are updated. When the policy immediate reward is maximized, the individual agent policy networks and the overall agent policy network are output for transformer fault early warning.

[0013] Furthermore, the immediate policy reward and the model loss of the overall agent policy network have the following relationship:

[0014] R t =-L M (θ M )

[0015] Among them, R t L represents the immediate reward for the t-th strategy. M (θ M θ represents the t-th model loss of the overall agent policy network. M This represents the model parameters of the overall agent policy network.

[0016] Furthermore, the model loss of the overall agent policy network is calculated as follows:

[0017] The model loss of the overall agent policy network is calculated using the following cross-entropy loss function:

[0018]

[0019] In the formula, L M () represents the model loss of the overall agent policy network. This represents the true label of the t-th data sample. This represents the t-th action of the overall agent policy network. Let π represent the t-th state of the overall agent policy network. M () represents the t-th policy of the overall agent policy network.

[0020] Furthermore, the parallel pre-training of single-agent policy networks corresponding to each attribute using the dataset includes:

[0021] Preprocess the dataset to obtain an intermediate dataset;

[0022] Feature extraction and feature filtering are performed on the intermediate dataset to determine the data features that meet the preset feature conditions as the training dataset;

[0023] The preset feature conditions include at least one of the following conditions (1) to (2):

[0024] (1) Feature similarity does not exceed the set threshold;

[0025] (2) Features are of the highest importance;

[0026] The policy networks of each individual agent are pre-trained in parallel using the training dataset.

[0027] Furthermore, for intermediate datasets with the same attribute, feature filtering of the intermediate datasets includes:

[0028] Extract the first data feature from the data samples in the intermediate dataset;

[0029] For the same data sample, calculate the similarity of its first data features, determine the first data feature that satisfies condition (1), and obtain the second data feature;

[0030] The importance of each second data feature is calculated using the random forest model, and the second data features that satisfy condition (2) are determined as the training dataset.

[0031] Furthermore, feature extraction from the intermediate dataset includes:

[0032] Extract time-domain and frequency-domain features from the intermediate dataset.

[0033] Furthermore, the properties of the transformer include at least one of temperature, sound, odor, and infrared data.

[0034] Secondly, the present invention provides a transformer fault early warning method, comprising:

[0035] Sensors are used to collect monitoring data on different properties of the transformer;

[0036] The monitoring data is input into the pre-trained single-agent policy networks according to the attribute categories to obtain the fault probability distribution. Each single-agent policy network corresponds one-to-one with each attribute of the transformer.

[0037] The fault probability distribution output by each individual agent policy network is input into the overall agent policy network to obtain the fault warning result.

[0038] Thirdly, the present invention provides a network training device, comprising:

[0039] The data acquisition unit is configured to acquire datasets of different attributes of the transformer collected by each sensor;

[0040] The network pre-training unit is configured to train each single agent policy network in parallel using the dataset based on the attributes of the transformer. Each single agent policy network corresponds one-to-one with each attribute of the transformer. The overall agent policy network is pre-trained based on the output of the pre-trained single agent policy networks.

[0041] The network training unit is configured to establish the relationship between the immediate policy reward and the network loss of the overall agent policy network. With the goal of maximizing the immediate policy reward, and using the output of the individual agent policy network as the input of the overall agent policy network, it collaboratively trains the individual agent policy networks and the overall agent policy network. It solves for the immediate policy reward based on the network loss of the overall agent policy network and updates the network parameters of the individual agent policy networks and the overall agent policy network. When the immediate policy reward is maximized, it outputs the individual agent policy networks and the overall agent policy network for transformer fault early warning.

[0042] Fourthly, the present invention provides a transformer fault early warning device, comprising:

[0043] The data acquisition unit is equipped with sensors corresponding to different data attributes and is configured to collect different data from the transformer.

[0044] A single agent computing unit, which is configured with a single agent policy network corresponding to different attributes, trained by the method provided by the first aspect or by the device of the third aspect, is configured to calculate the fault probability distribution based on monitoring data.

[0045] The overall intelligent agent computing unit, which is equipped with an overall intelligent agent policy network, is trained by the method provided by the first aspect or by using the device of the third aspect, and is configured to calculate the fault warning result according to the fault probability distribution.

[0046] Fifthly, the present invention provides a multi-agent system, comprising a total agent and at least one single agent;

[0047] A single agent processes the data from the sensor through a single agent policy network and outputs a fault probability distribution. The total agent processes the fault probability distribution output from the single agent through a total agent policy network and obtains the second fault warning result.

[0048] The single-agent policy network and the overall agent policy network are trained by the method provided by the first aspect or by the device of the third aspect.

[0049] In a sixth aspect, the present invention provides an electronic device, comprising: a memory and a processor;

[0050] The memory is used to store programs;

[0051] The processor is configured to invoke a program stored in the memory to execute the multi-agent policy network training method provided in the first aspect embodiment and / or any possible implementation in combination with the first aspect embodiment, or to execute the transformer fault early warning method provided in the second aspect embodiment.

[0052] In a seventh aspect, the present invention also provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the multi-agent policy network training method provided in the first aspect embodiment and / or in combination with any possible implementation of the first aspect embodiment, or executes the transformer fault early warning method provided in the second aspect embodiment.

[0053] As can be seen from the above technical solutions, the present invention has the following beneficial effects:

[0054] This invention provides a multi-agent policy network training method, a transformer fault early warning method, and a device. The training method first pre-trains individual agent policy networks and the overall agent policy network in parallel. Then, it establishes the relationship between the immediate reward of the agent policy control and the network loss of the overall agent policy network. Next, it updates the network with the goal of maximizing the immediate reward; this update process is essentially adjusting the network parameters. The output actions of the individual agent policy networks are used as the input to the overall agent policy network. By maximizing the immediate reward, the optimal network parameters are determined, thus completing the training of the agent policy network. The policy network obtained through this training method allows the agents to fully utilize information from multiple sensors. Through deep reinforcement learning among multiple agents, mutual cooperation and information fusion are achieved, thereby realizing comprehensive early warning and control of power transformer systems. Implementing this invention improves the accuracy and effectiveness of early warning and adapts to complex power system environments. Furthermore, the parallel pre-training of the policy networks accelerates the overall training process, enabling real-time data processing and decision-making. The agents learn about the dynamic changes in the environment and make corresponding early warning and control decisions based on the current state and environmental feedback, thereby enhancing the system's real-time performance and adaptability. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a structural block diagram of a multi-agent system 100 provided in an embodiment of the present invention;

[0057] Figure 2 This is a structural block diagram of a multi-agent system 200 for transformer fault early warning provided in an embodiment of the present invention;

[0058] Figure 3 The implementation flow of a multi-agent policy network training method provided in this embodiment of the invention;

[0059] Figure 4 The present invention provides an implementation flow of a multi-agent policy network training method using a transformer fault early warning model as the training result.

[0060] Figure 5 The following is an implementation flow of a transformer fault early warning method provided in an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the structure of a network training device provided in an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of the structure of a transformer fault early warning device provided in an embodiment of the present invention;

[0063] Figure 8 This is an electronic device architecture diagram provided for an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Traditional transformer fault monitoring technologies are limited by the inefficiency and high cost of manual inspections. Their processing and decision-making based on sensor data rely on pre-defined rules and thresholds, lacking flexibility and adaptability. Furthermore, the multivariable and nonlinear characteristics of power transformer systems make traditional monitoring technologies ill-suited to complex power system changes and fault modes. In addition, commonly used methods often employ simplified models and assumptions, failing to adequately consider the system's dynamic characteristics and potential nonlinear relationships, potentially leading to poor accuracy and effectiveness in early warning and control. The lengthy data processing and decision-making processes also result in slow response times, while rapid identification and response to equipment faults are crucial in power systems.

[0066] Intelligent agents are currently a major research focus in deep reinforcement learning and artificial intelligence. Generally, an intelligent agent is a computational entity that can autonomously adjust its behavior and state according to changes in its environment, possessing characteristics such as initiative, responsiveness, and sociality. In many real-world scenarios, people need to control multiple intelligent agents simultaneously to complete specific tasks. Therefore, research on reinforcement learning has gradually expanded from the single-agent domain to multi-agent reinforcement learning (MARL).

[0067] In MARL, a policy controls multiple agents to interact with the environment simultaneously, with the goal of maximizing the cumulative reward. Here, the transitions of the global state and the reward are related to the joint actions of all agents, rather than the actions of individual agents. Therefore, during policy learning, each agent's policy update needs to consider the current policies of other agents.

[0068] Multi-agent scenarios are more complex than single-agent scenarios because each agent interacts with the environment while also directly or indirectly interacting with other agents. Since multiple agents interact dynamically in real-time within the environment, and each agent is constantly learning and updating its own policy, the environment is non-stationary from each agent's perspective. That is, for an agent, even if it takes the same action in the same state, the resulting state transitions and reward signal distributions may constantly change. Furthermore, multi-agent reinforcement learning may be multi-objective, with different agents needing to maximize their own interests. This increases the complexity of training and evaluation, potentially requiring large-scale distributed training to improve efficiency.

[0069] Explanation of terms used in the embodiments of this invention:

[0070] Intelligent agent: A computational entity that can make subjective judgments and perform actions based on experience.

[0071] Environment: The scene in which the agent performs actions, which can interact with the agent and be affected by the actions taken by the agent, and provides feedback on the status and rewards to the agent.

[0072] State: An agent's understanding and encoding of the environment, which usually contains information that affects the decisions made by the agent.

[0073] Action: An action performed by an agent in response to the current state of the environment. It often refers to a conceptual action, and in some cases also includes the mechanism that executes the action.

[0074] Policy: refers to the basis upon which an agent performs a certain action in a given state. That is, given a state, an agent can choose the action to take based on a policy.

[0075] Reward: The benefit an agent receives from the environment after taking a series of actions. It usually includes a combination of rewards and punishments in a real sense. Rewards are generally represented by positive values ​​and punishments by negative values.

[0076] To address the aforementioned problems, this invention provides an agent policy network training method, a transformer fault early warning method, and a device through the following embodiments. By learning through interaction with the environment, the decision-making strategy of the agent is optimized, and a comprehensive early warning model is constructed to achieve comprehensive monitoring and fault early warning of the power transformer system.

[0077] See Figure 1 One embodiment of the present invention provides a multi-agent system 100, including a total agent 110 and at least one single agent 120.

[0078] Both the overall agent 110 and at least one single agent 120 can interact with the environment and output decision actions according to the state of the environment through their respective policies.

[0079] In the embodiments provided by the present invention, a single agent processes data from the environment through a single agent policy network and outputs a first action, and a total agent processes the first action output from the single agent through a total agent policy network and outputs a second action.

[0080] Specifically, the single agent includes a first acquisition module, which processes the state acquired from the environment to obtain the action to be performed by the single agent; the total agent includes a second acquisition module, which processes the state acquired from the environment and the action performed by the single agent to obtain the action to be performed by the total agent.

[0081] The actions received by the overall intelligent agent can come from a single intelligent agent or from multiple single intelligent agents; this embodiment does not limit this.

[0082] The interaction between the overall intelligent agent and individual intelligent agents includes, but is not limited to: the actions of the intelligent agent itself, control commands sent to other intelligent agents, and content expressed in machine language using a fixed data structure.

[0083] When referring to the environment, in addition to the physical environment of the scene in which the intelligent agent is located, it can also refer to other intelligent agents or devices in the same scene as the intelligent agent. For example, intelligent agents that also make decisions and are configured with strategies based on changes in the physical environment, or devices that can interact with the intelligent agent other than those mentioned above, such as servers, sensors, and terminals in the physical environment.

[0084] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0085] The aforementioned sensors refer to devices that can detect events or changes in the environment and send this message to other electronic devices (such as central processing units), and can react through feedback based on the properties of things.

[0086] The term "terminal" as mentioned above refers to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. It can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0087] For example, Figure 2 A multi-agent system 200 for transformer fault early warning is shown, including a total agent 210 and four single agents 221-224, namely a temperature agent 221, a sound agent 222, an odor agent 223, and an infrared agent 224.

[0088] Individual agents 221-224 interact with temperature sensor 231, sound and vibration sensor 232, odor sensor 233, and infrared sensor 234 in the physical environment, respectively, and send their actions to the overall agent 210. In this embodiment, the interaction process between the agents and the environment can be understood as follows: individual agents observe the monitoring data of various attributes of the transformer, select actions to execute based on their respective strategies, and the overall agent observes the actions executed by each individual agent and selects actions to execute based on its own strategy.

[0089] When discussing transformer fault early warning, the action executed by the agent is usually a vector, which refers to the probability distribution of the fault or a binary judgment of the occurrence of the fault. For example, the action executed by temperature agent 221 represents the probability distribution of transformer faults obtained from temperature monitoring data, or the judgment result of transformer fault and normal operation.

[0090] See Figure 3 An embodiment of the present invention provides a multi-agent policy network training method. The multi-agent policy network trained by this method is used for transformer fault early warning. The method includes:

[0091] Step S31. Obtain the dataset of different attributes of the transformer collected by each sensor.

[0092] Specifically, different sensors collect data on different properties of the transformer. For example, a temperature sensor collects the temperature information of the transformer, a sound and vibration sensor collects the audio information of the transformer, an odor sensor collects the odor information of the transformer, and an infrared sensor collects the infrared image of the transformer.

[0093] In some embodiments, step S31 may also include preprocessing of the data, such as removing outliers, missing values, normalization, standardization, etc.

[0094] Step S32. Based on the properties of the transformer, train each single agent policy network in parallel using the dataset. Each single agent policy network corresponds one-to-one with each property of the transformer. Pre-train the overall agent policy network based on the output of the pre-trained single agent policy networks.

[0095] Specifically, each single-agent policy network corresponds to an attribute of a transformer; that is, datasets with the same attribute are input into the same single-agent policy network. The single-agent policy network can learn the feature representations and decision-making strategies for the fault warning task from labeled data samples. The pre-training stage provides a good starting point for better policy optimization and learning in subsequent reinforcement learning processes. After pre-training, reinforcement learning algorithms can be used to further optimize the agent's policy network and perform online learning and decision-making in a real-world environment.

[0096] The overall agent policy network takes the output of the single agent policy network as input to perform the calculation and learning of the decision-making process, and the output layer outputs the action policy.

[0097] For example, a single-agent policy network typically includes an input layer, hidden layers, and an output layer. The input layer receives the dataset as input, the hidden layer can use multiple fully connected layers or convolutional layers for feature extraction and representation learning, and the output layer can output the probability distribution of the warning classification, representing the confidence level of normal and fault, or the binary judgment result of normal and fault.

[0098] In some embodiments, before the dataset is input into the single-agent policy network, it needs to be preprocessed and feature-selected to extract useful features from the data samples, reduce data redundancy and noise, improve training effect and generalization ability. At the same time, feature selection before inputting the network for training can reduce data dimensionality, improve network model performance, accelerate the training process, and reduce model complexity and computational cost of the training process.

[0099] In some embodiments, step S32, which involves training the policy networks of each single agent in parallel using the dataset based on the properties of the transformer, includes the following steps:

[0100] Step S321. Preprocess the dataset to obtain an intermediate dataset.

[0101] For example, preprocessing includes preprocessing the data, such as removing outliers, missing values, normalizing, and standardizing.

[0102] Step S322. Perform feature extraction and feature filtering on the intermediate dataset to determine the data features that meet the preset feature conditions as the training dataset.

[0103] The preset feature conditions include at least one of the following conditions (1) to (2):

[0104] (1) The feature similarity does not exceed the first set threshold;

[0105] (2) Features are of the highest importance.

[0106] Specifically, feature similarity screening refers to extracting multiple features of the same attribute for the same data sample, comparing the similarity of these features, excluding highly similar features, and avoiding data redundancy.

[0107] For example, taking a data sample containing sound attributes, we can extract the spectral features, sound intensity features, and sound variation features of a data sample, compare the similarity of these features, and exclude those with high similarity. Alternatively, we can use a similarity threshold for filtering.

[0108] In some examples, similarity comparisons can be performed using metrics such as Euclidean distance, cosine similarity, Pearson correlation coefficient, and Manhattan distance.

[0109] The importance of a feature reflects the degree to which a data feature influences the actions performed by an agent. In some examples, this invention uses a random forest model to calculate the importance of each data feature and selects several data features with higher importance as inputs to a single-agent policy network.

[0110] Step S323. Use the training dataset to pre-train the policy networks of each single agent in parallel.

[0111] Specifically, the data features of the data samples obtained through feature extraction and feature filtering are used as input to pre-train the policy networks of each single agent in parallel. For example, the policy network of the temperature agent is trained using the data features of the temperature data, and the policy network of the sound agent is trained using the data features of the sound vibration data. The policy networks of each agent complete the pre-training process in parallel.

[0112] In some embodiments, the pre-trained total agent policy network based on the output of the pre-trained single agent policy network in step S32 includes:

[0113] The input layer is the action representation output by the single-agent policy network corresponding to attributes such as temperature, sound vibration, odor, and infrared image. The hidden layer uses multiple fully connected layers to calculate and learn the decision-making process. The output layer outputs the decision action, which is usually a vector representing the fault probability distribution or direct action decision obtained by integrating the data of various attributes.

[0114] In some embodiments, stochastic gradient descent is used during the pre-training phase to minimize the network loss of the overall agent policy network.

[0115] Step S33. Establish the relationship between the policy immediate reward and the network loss of the total agent policy network.

[0116] Step S34. With the goal of maximizing the immediate reward of the policy, and using the output of the single agent policy network as the input of the overall agent policy network, train the single agent policy networks and the overall agent policy network in a collaborative manner.

[0117] Step S35. Solve for the policy immediate reward based on the network loss of the overall agent policy network and update the network parameters of each individual agent policy network and the overall agent policy network. When the policy immediate reward is maximized, output the individual agent policy networks and the overall agent policy network used for transformer fault early warning.

[0118] Specifically, the embodiments provided by the present invention represent the immediate reward of the policy through the network loss of the overall agent policy network. During the training phase, the network parameters of each individual agent policy network and the overall agent policy network are updated to calculate the network loss of the overall agent policy network. Since there is a mapping relationship between the network parameters and the network loss, the solution for maximizing the immediate reward of the policy can be found based on the relationship established in step S33. The network parameters that maximize the immediate reward of the policy can be sought, that is, the variable values ​​corresponding to the optimal solution of the objective function.

[0119] For example, such as Figure 4 As shown, one embodiment of the present invention provides a method for training an agent policy network using a transformer fault early warning model as the training result, comprising the following steps:

[0120] Step S41. Obtain data for each attribute of the transformer.

[0121] Specifically, sensors or monitoring equipment are used to monitor the various properties of the transformer in real time and record data, including temperature, sound vibration, and infrared data.

[0122] Step S42. Data preprocessing.

[0123] Specifically, the collected data is cleaned and preprocessed, including removing outliers and handling missing values, and normalizing or extracting features from the data.

[0124] Step S43. Perform feature extraction and feature filtering on the preprocessed data from step S42.

[0125] Specifically, feature extraction is mainly divided into time-domain features and frequency-domain features. For example, obtaining the spectral information of a signal, including the main frequency and spectral energy distribution, can be used to detect fault frequencies or abnormal patterns.

[0126] For example, the data characteristics of data collected by a sound sensor may include: sound spectrum characteristics, sound intensity characteristics, sound change characteristics, etc.; the data characteristics of data collected by a temperature sensor may include: average temperature characteristics, temperature change characteristics, temperature distribution characteristics, etc.; the data characteristics of data collected by an infrared sensor may include: heat map characteristics, temperature gradient characteristics, hot spot change characteristics, etc.

[0127] Feature selection employs a combination of feature similarity and random forest-based methods for more comprehensive and effective feature selection. The specific process is as follows: First, feature similarity analysis is performed, calculating the similarity between features and eliminating highly similar features, retaining only one to avoid data redundancy. Then, feature selection is based on random forests, i.e., training a random forest model, using the random forest model to evaluate the importance score of each feature, and selecting features with higher importance based on their importance scores.

[0128] Step S44. Initialize the single-agent policy networks corresponding to temperature, sound, and infrared data, and the master agent policy network.

[0129] Specifically, the state variables of each agent are defined as follows:

[0130] Temperature-sensitive agent: Indicates the current temperature of the transformer;

[0131] Voice-based intelligent agents: This indicates the current acoustic vibration signal of the transformer;

[0132] Infrared intelligent agent: Indicates the current infrared status of the transformer;

[0133] Main agent: This represents the actions and data characteristics of each individual agent.

[0134] The actions of each agent are defined as follows:

[0135] Temperature-sensitive agent: This represents the probability of failure calculated based on the current temperature of the transformer.

[0136] Voice-based intelligent agents: This represents the probability of failure calculated based on the current acoustic and vibration signals of the transformer.

[0137] Infrared intelligent agent: This represents the probability of failure calculated based on the current infrared status of the transformer.

[0138] Main agent: This indicates the fault diagnosis result, i.e., whether the transformer is faulty or normal.

[0139] The strategies for each agent are defined as follows:

[0140] Temperature-sensitive agent:

[0141] Voice-based intelligent agents:

[0142] Infrared intelligent agent:

[0143] Main agent:

[0144] And define the target label of the data sample as This represents the actual fault condition. In some embodiments, binary values ​​can be used to represent fault and normal, for example, 0 represents normal and 1 represents fault.

[0145] Each single-agent policy network adopts the same network structure, and the output layer outputs a fault probability distribution, representing the confidence level of transformer fault and normal operation.

[0146] The loss function for each single-agent policy network is calculated using the cross-entropy loss function, which is defined as follows:

[0147] Temperature-sensitive agent policy network:

[0148] Voice agent policy network:

[0149] Infrared agent policy network:

[0150] Where, θ T θ S θ I These represent the network parameters of each individual agent's policy network.

[0151] Step S45. Pre-train the policy networks of each single agent in parallel.

[0152] Specifically, a single-agent policy network corresponding to temperature, acoustic vibration, and infrared data is iteratively pre-trained using supervised learning.

[0153] Step S46. Pre-train the overall agent policy network based on the actions of each pre-trained individual agent policy network.

[0154] To accelerate the training speed of the overall agent policy network, a supervised model is used for pre-training.

[0155] Specifically, the network structure of the overall agent policy network is as follows: the input layer is the action representation output by each individual agent policy network, the hidden layer uses multiple fully connected layers to calculate and learn the decision-making process, and the output layer outputs the action policy, which is usually a vector representing the probability distribution of each possible action or the direct action selection.

[0156] The loss function of the overall agent policy network is defined as follows:

[0157] θ M This represents the model parameters of the overall agent policy network.

[0158] Step S46. Define the objective function for updating the policy networks of each individual agent and the overall agent policy network.

[0159] Specifically, during formal training, the network parameters of each agent's policy network are adjusted to solve the objective function, which is defined as:

[0160] J(θ r ) = LM (θ r ), r∈(T,S,I,M)

[0161] Update the network parameters to minimize the objective function.

[0162] The immediate reward of the strategy is defined as R. t =-L M (θ M The smaller the network loss, the greater the immediate reward of the policy. By updating the network parameters, the immediate reward of the policy is maximized, thereby approaching the optimal policy.

[0163] Step S47. Using the output of the single agent policy network as the input of the overall agent policy network, train the single agent policy networks and the overall agent policy network collaboratively.

[0164] Step S48. Output the trained individual agent policy networks and the overall agent policy network.

[0165] Specifically, the individual agent policy networks and the overall agent policy network together constitute the transformer fault early warning model.

[0166] The training method provided in the above embodiments allows multiple agents to autonomously learn and adapt to complex power system changes and fault modes through deep reinforcement learning algorithms. This overcomes the limitations of rules and threshold settings in traditional methods. By learning through interaction with the environment, decision-making strategies are optimized, no longer relying on fixed rules and thresholds. Considering data from multiple sensors, the deep reinforcement learning method can fully utilize the information from each sensor, improving the accuracy and effectiveness of early warnings. Collaboration and information fusion among agents enable the early warning model to more comprehensively monitor the power transformer system and provide more accurate fault warnings through agent decision optimization. Parallel pre-training accelerates the training process, enabling real-time data processing and decision-making. By learning the dynamic changes in the environment and making corresponding early warning and control decisions based on the current state and environmental feedback, the agents enhance the system's real-time performance and adaptability.

[0167] See Figure 5 One embodiment of the present invention provides a transformer fault early warning method, comprising the following steps:

[0168] Step S51. Use sensors to collect monitoring data of different properties of the transformer.

[0169] Step S52. Input the monitoring data into the trained single agent policy network according to the attribute category to obtain the fault probability distribution. Each single agent policy network corresponds one-to-one with each attribute of the transformer.

[0170] Step S53. Input the fault probability distribution output by each individual agent policy network into the overall agent policy network to obtain the fault warning result.

[0171] The above-disclosed embodiments describe in detail an agent policy network training method. This training method can be implemented using various types of devices. Therefore, the present invention also discloses a network training device corresponding to the above method. Specific embodiments are given below for detailed description.

[0172] like Figure 6 As shown, one embodiment of the present invention provides a network training apparatus 600, comprising:

[0173] The data acquisition unit 610 is configured to acquire datasets of different attributes of the transformer collected by each sensor;

[0174] The network pre-training unit 620 is configured to train each single agent policy network in parallel using a dataset based on the attributes of the transformer. Each single agent policy network corresponds one-to-one with each attribute of the transformer. The overall agent policy network is pre-trained based on the output of the pre-trained single agent policy networks.

[0175] The network training unit 630 is configured to establish the relationship between the immediate policy reward and the network loss of the overall agent policy network. With the goal of maximizing the immediate policy reward, and using the output of the individual agent policy network as the input of the overall agent policy network, it collaboratively trains the individual agent policy networks and the overall agent policy network. It solves for the immediate policy reward based on the network loss of the overall agent policy network and updates the network parameters of the individual agent policy networks and the overall agent policy network. When the immediate policy reward is maximized, it outputs the individual agent policy networks and the overall agent policy network for transformer fault early warning.

[0176] The above-disclosed embodiments describe in detail a transformer fault early warning method. The above-disclosed method can be implemented using various types of devices. Therefore, the present invention also discloses a transformer fault early warning device corresponding to the above method. Specific embodiments are given below for detailed description.

[0177] like Figure 7 As shown, one embodiment of the present invention provides a transformer fault early warning device 700, comprising:

[0178] The data acquisition unit 710 is equipped with sensors corresponding to different data attributes and is configured to acquire different data from the transformer.

[0179] The single agent computing unit 720 is configured with a single agent policy network corresponding to different attributes and is configured to calculate the fault probability distribution based on monitoring data.

[0180] The total intelligent agent computing unit 730 is configured with a total intelligent agent policy network, which is configured to calculate the fault warning result based on the fault probability distribution.

[0181] The network training device 600 and transformer fault early warning device 700 provided in this application embodiment have the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0182] See Figure 8 It shows a hardware structure block diagram of an electronic device, including: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0183] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0184] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0185] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0186] The memory stores a program, which the processor can call. The program is used to: implement the various processing steps of the training method of the aforementioned improved prototype network, or implement the various processing steps of the aforementioned sound classification method.

[0187] This invention also provides a readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the training method of the improved prototype network provided in any possible implementation of the above embodiments and / or in combination with the embodiments, or executes the sound classification method provided in the above embodiments.

[0188] It should be understood that although the steps in the various figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in each figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0189] Those skilled in the art will understand that the structures shown in the figures are merely block diagrams of some structures related to the present application and do not constitute a limitation on the terminal device to which the present application is applied. Specific terminal devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0190] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0191] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0192] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a multi-agent policy network, wherein the agent policy network trained by the method is used for transformer fault early warning, characterized in that, The method includes: Acquire a dataset of different attributes of the transformer collected by each sensor, wherein the attributes include at least one of the transformer's temperature information, audio information, odor information, and infrared image; Based on the properties of the transformer, the policy networks of each single agent are trained in parallel using the dataset. Each single agent policy network corresponds one-to-one with each property of the transformer. The overall agent policy network is pre-trained based on the output of the pre-trained single agent policy networks. Establish the relationship between immediate policy rewards and the network loss of the overall agent policy network; With the goal of maximizing immediate policy rewards, and using the output of the single agent policy network as the input of the overall agent policy network, we collaboratively train the individual agent policy networks and the overall agent policy network. The policy immediate reward is calculated based on the network loss of the overall agent policy network, and the network parameters of each individual agent policy network and the overall agent policy network are updated. When the policy immediate reward is maximized, the individual agent policy networks and the overall agent policy network are output for transformer fault early warning.

2. The multi-agent policy network training method according to claim 1, characterized in that, The immediate reward of the policy and the model loss of the overall agent policy network have the following relationship: R t =-L M (i M ) Among them, R t L represents the immediate reward for the t-th strategy. M (θ M θ represents the t-th model loss of the overall agent policy network. M This represents the model parameters of the overall agent policy network.

3. The multi-agent policy network training method according to claim 1, characterized in that, The method of training each single-agent policy network in parallel using a dataset based on the properties of the transformer includes: The dataset is preprocessed to obtain an intermediate dataset; Feature extraction and feature filtering are performed on the intermediate dataset to determine data features that meet preset feature conditions as the training dataset; The preset feature conditions include at least one of the following conditions (1) to (2): (1) Feature similarity does not exceed the set threshold; (2) Features are of the highest importance; The policy networks of each single agent are pre-trained in parallel using the training dataset.

4. The multi-agent policy network training method according to claim 3, characterized in that, For intermediate datasets with the same attribute, the feature filtering process for the intermediate datasets includes: Extract the first data feature from the data samples in the intermediate dataset; For the same data sample, calculate the similarity of its first data features, determine the first data feature that satisfies condition (1), and obtain the second data feature; The importance of each second data feature is calculated using the random forest model, and the second data features that satisfy condition (2) are determined as the training dataset.

5. A method for early warning of transformer faults, characterized in that, include: Sensors are used to collect monitoring data on different properties of the transformer; The monitoring data is input into the single agent policy networks trained by the method described in any one of claims 1 to 4 according to the category of the attribute to obtain the fault probability distribution. Each single agent policy network corresponds one-to-one with each attribute of the transformer. The fault probability distribution output by each individual agent policy network is input into the overall agent policy network to obtain the fault warning result.

6. A network training device, characterized in that, include: The data acquisition unit is configured to acquire a dataset of different attributes of the transformer collected by each sensor, the attributes including at least one of the transformer's temperature information, audio information, odor information, and infrared image; The network pre-training unit is configured to train each single agent policy network in parallel using the dataset based on the attributes of the transformer. Each single agent policy network corresponds one-to-one with each attribute of the transformer. The overall agent policy network is pre-trained based on the output of the pre-trained single agent policy networks. The network training unit is configured to establish the relationship between the immediate policy reward and the network loss of the overall agent policy network. With the goal of maximizing the immediate policy reward, and using the output of the individual agent policy network as the input of the overall agent policy network, it collaboratively trains the individual agent policy networks and the overall agent policy network. It solves for the immediate policy reward based on the network loss of the overall agent policy network and updates the network parameters of the individual agent policy networks and the overall agent policy network. When the immediate policy reward is maximized, it outputs the individual agent policy networks and the overall agent policy network for transformer fault early warning.

7. A transformer fault early warning device, characterized in that, include: The data acquisition unit is equipped with sensors corresponding to different data attributes and is configured to collect different data from the transformer. A single agent computing unit, wherein a single agent policy network corresponding to different attributes is configured, trained by the method as described in any one of claims 1 to 4 or by the device as described in claim 6, and is configured to calculate the fault probability distribution based on monitoring data; A total agent computing unit, wherein a total agent policy network is configured, trained by the method as described in any one of claims 1 to 4 or by the apparatus as described in claim 6, is configured to calculate fault warning results based on the fault probability distribution.

8. A multi-agent system, characterized in that, It includes the total intelligent agent and at least one single intelligent agent; The single agent processes the data from the sensor through the single agent policy network and outputs a fault probability distribution. The total agent processes the fault probability distribution output from the single agent through the total agent policy network to obtain a fault warning result. The single-agent policy network and the total agent policy network are obtained by the method described in any one of claims 1 to 4 or by training using the apparatus described in claim 6.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to call a program stored in the memory to execute the multi-agent policy network training method as described in claims 1 to 4, or to execute the transformer fault early warning method as described in claim 5.

10. A readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the multi-agent policy network training method as described in claims 1 to 4, or executes the transformer fault early warning method as described in claim 5.

Images