Power equipment anomaly detection method based on audio text cross-modal multi-loss fusion network

Abstract

The invention relates to a power equipment anomaly detection method based on an audio text cross-modal multi-loss fusion network, and belongs to the field of power equipment operation anomaly detection. And designing a fusion network for realizing fusion of multi-modal data, and training by adopting a multi-loss training mechanism to realize anomaly detection of the power equipment. According to the method, audio and text multi-mode data are utilized, the accuracy and comprehensiveness of power equipment fault detection are improved, and the cost of manual inspection and regular maintenance can be reduced.

Description

Technical Field

[0001] The present invention belongs to the field of power equipment operation anomaly detection, and specifically relates to a power equipment anomaly detection method based on an audio-text cross-modal multi-loss fusion network. Background Art

[0002] The stable operation of power equipment is the cornerstone of power system safety, and its importance is self-evident. During the operation of the power system, power equipment is under high load and complex environment for a long time, and various abnormal conditions will inevitably occur, such as equipment failure, aging, insulation damage, etc. If these abnormal conditions are not discovered and properly handled in time, they may trigger a series of chain reactions, which may cause damage to power equipment and affect the stability of power supply. In severe cases, they may cause power system failures and even lead to serious safety accidents, posing a huge threat to people's lives and property safety.

[0003] At present, power equipment anomaly detection mainly adopts manual inspection and regular maintenance. Manual inspection requires staff to go to the site to check the equipment in person. This method is not only inefficient and consumes a lot of manpower and time costs, but also easily affected by human factors, and the accuracy and reliability of the detection results are difficult to guarantee. In addition, manual inspection can only be carried out at a specific time, and it is impossible to monitor the equipment in real time, making it difficult to find some potential problems. Regular maintenance is to maintain and inspect the equipment at a certain time interval. Although it can prevent equipment failure to a certain extent, this method is often based on experience and conventional detection methods, and may not be able to detect some sudden and hidden abnormal situations in time.

[0004] With the continuous development and complexity of the power system, the requirements for power equipment anomaly detection technology are getting higher and higher. Traditional detection methods can no longer meet the needs of modern power systems. Therefore, there is an urgent need for a more efficient and accurate power equipment anomaly detection technology that can detect abnormal conditions of equipment in real time and accurately, issue early warning signals in a timely manner, and provide strong support for equipment maintenance and repair, thereby ensuring the safe and stable operation of the power system. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the existing power equipment fault detection method and provide a power equipment anomaly detection method based on an audio-text cross-modal multi-loss fusion network.

[0006] To achieve the above-mentioned purpose, the technical solution of the present invention is: a method for detecting anomalies of power equipment based on an audio-text cross-modal multi-loss fusion network, a fusion network designed to realize the fusion of multi-modal data, and a multi-loss training mechanism is used for training to realize anomaly detection of power equipment.

[0007] In one embodiment of the present invention, before multimodal fusion of data, feature extraction is performed through a feature network.

[0008] In one embodiment of the present invention, the feature network includes a pre-trained text feature network and an audio feature network, the text feature network uses RoBERTa, and the audio feature network uses a long short-term memory network.

[0009] In one embodiment of the present invention, the fusion network includes a cross attention encoder, a self-attention encoder and a point-by-point feedforward network, wherein:

[0010] The criss-cross attention encoder adopts a mechanism similar to the self-attention encoder, using queries from one model and keys and values ​​from another model to capture the interdependencies between different modalities;

[0011] A self-attention encoder is used to model the connections across time steps of the new feature representation after passing through the criss-cross attention encoder, thereby providing informative input to the point-by-point feedforward network;

[0012] A point-by-point feedforward network accesses the information from the self-attention encoder, applies a fully connected feedforward network and ReLU activation function to each individual position, and refines the encoded feature representation.

[0013] In one embodiment of the present invention, the cross-attention encoder is described by the following formula:

[0014]

[0015] in:

[0016] Q m1 =W q ·f m1

[0017] K m2 =W k ·f m2

[0018] V m2 =W v ·f m2

[0019] The query is represented as Q m1 , which comes from Model 1; denote the key and value as K m2 and V m2 , which comes from model 2; f m1 is the feature of model 1, f m2 is the characteristic of model 2, W q , W k W v Corresponding to different features f m1、f m2 The weight of is the visual modality feature, d k is the scaling factor.

[0020] In one embodiment of the present invention, a multi-loss training mechanism is used for training, which is specifically implemented as follows:

[0021] An additional fully connected layer is added between the text feature network and the fusion network, and between the audio feature network and the fusion network; three different loss functions are applied during training, each corresponding to an output, and the final loss is the sum of the loss from the subnet and the loss from the fusion network:

[0022] Loss = ∑α m *lossfunction(y m ,target m )

[0023] Among them, α m is the weight parameter, lossfunction is the loss function equation, y m is the output calculated by the network, target m is the target label.

[0024] In one embodiment of the present invention, an original signal recovery variant designed to mitigate the loss of the original signal during the cross-model projection process is also included.

[0025] In one embodiment of the present invention, the original signal recovery variant includes:

[0026] Concatenation variant, concatenating the original features with the fused features;

[0027] Transformer variant that merges the original hidden state and the fused hidden state and processes the merged hidden state using the Transformer encoder;

[0028] The combined features are passed through a linear layer and concatenated with features from other modes, maintaining the integrity of the original signal while integrating cross-model information.

[0029] The present invention also provides an electric power equipment anomaly detection system based on an audio-text cross-modal multi-loss fusion network, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0030] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Improve the accuracy of fault detection: By integrating audio and text multimodal data, making full use of the sound features in the audio information and the detailed information such as fault evaluation, noise interference, subsequent processing methods and the location of the equipment fault components in the text information, it is possible to more comprehensively and accurately judge the fault conditions of power equipment and improve the accuracy of fault detection.

[0033] 2. Comprehensively understand the operating status of equipment: Comprehensive consideration of audio and text information can provide a more comprehensive understanding of the operating status of power equipment, including fault type, location, severity, etc., providing a more accurate basis for equipment maintenance and management.

[0034] 3. Improve the intelligence level of fault diagnosis: Use advanced models and algorithms for multimodal training to make fault diagnosis more intelligent, able to automatically learn and identify complex fault modes, and improve the accuracy and reliability of diagnosis.

[0035] 4. Enhance preventive maintenance capabilities: Through real-time monitoring and accurate fault diagnosis, preventive maintenance measures can be taken in time to avoid further deterioration of faults and extend the service life of equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a cross attention fusion network.

[0037] Figure 2 Schematic diagram of original signal recovery. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0041] The present invention provides a method for detecting anomalies of power equipment based on an audio-text cross-modal multi-loss fusion network. The method designs a fusion network for realizing fusion of multi-modal data and adopts a multi-loss training mechanism for training to realize anomaly detection of power equipment.

[0042] The following is the specific implementation process of the present invention.

[0043] like Figure 1 As shown, this embodiment provides a method for abnormality detection of power equipment based on an audio-text cross-modal multi-loss fusion network, and the technical route is as follows: first, a feature network is constructed, and pre-trained text and audio processing models are used to extract features; second, a fusion network is designed, including a cross-attention encoder, a self-attention encoder, and a point-by-point feedforward network, to achieve the fusion of multimodal information; then, multi-loss training is performed, additional fully connected layers are added to the fusion network architecture, and multiple loss functions are applied to improve model performance and generalization ability; then, an original signal recovery variant is developed to reduce the loss of the original signal in the cross-model projection process; finally, context modeling methods are explored to integrate context data into the model to enhance the accuracy of anomaly detection.

[0044] The specific steps proposed by the present invention are as follows:

[0045] Step 1: Feature network construction.

[0046] Two different pre-trained text and audio processing models are used. The text feature network uses RoBERTa, which is selected for its significantly superior performance on various downstream tasks. The audio feature network uses a long short-term memory network, which can better handle long-term dependencies in long sequence data and has a good performance in extracting audio features.

[0047] Step 2: Converged network construction.

[0048] The fusion network is the core of the multi-modal multi-loss fusion network, such as Figure 1 As shown in the yellow part, it consists of a cross-attention encoder, a self-attention encoder, and a point-by-point feedforward network.

[0049] The Cross-Attention Encoder uses a mechanism similar to the Self-Attention Encoder, using queries from one model and keys and values ​​from another model to capture the interdependencies between different modalities. For example, text features are used to construct queries for text and also for constructing keys and values ​​for audio.

[0050] The criss-cross attention encoder can be described as follows:

[0051]

[0052] in:

[0053] Q m1 =W q ·f m1

[0054] K m2 =W k ·f m2

[0055] V m2 =W v ·f m2

[0056] The query is represented as Q m1 , which comes from Model 1; denote the key and value as K m2 and V m2 , which comes from model 2; f m1 is the feature of model 1, f m2 is the characteristic of model 2. W q , W k W v Corresponding to different features f m1 、f m2 The weight of is the visual modality feature, d k is the scaling factor.

[0057] Figure 1 In the text feature f t The query q used to construct the text t , which is also used to construct the audio keyword k a Sum value v a The cross-attention encoder essentially projects the hidden state of one model into the space of another model.

[0058] A self-attention encoder is used to model the connections across time steps of the new feature representation after passing through the criss-cross attention encoder, thereby providing informative input to the point-by-point feedforward network;

[0059] A point-by-point feedforward network accesses the information from the self-attention encoder, applies a fully connected feedforward network and ReLU activation function to each individual position, and refines the encoded feature representation.

[0060] Step 3: Multi-loss training.

[0061] In the architecture of the fusion network, an additional fully connected layer is formed at the terminal of each feature network. This modification supports two additional outputs from independent modes in addition to the combined feature output. Three different loss functions are applied during training, one for each output. The final loss is the sum of the losses from the subnetworks and the loss from the fusion network:

[0062] Loss = ∑α m *lossfunction(y m ,target m

[0063] Among them, α m is the weight parameter, lossfunction is the loss function equation, y m is the output calculated by the network, target m is the target label.

[0064] The purpose of the multi-task loss is to support the understanding and processing of each feature network on its own signal, encourage each feature network to improve its method of processing a specific model, and effectively combine the different signals transmitted by the feature networks under the guidance of the training fusion network.

[0065] Step 4: Original signal restoration.

[0066] Two fusion network variants are developed to mitigate the potential loss of original signals during cross-model projection, e.g. Figure 2 Shown in the crimson color part above.

[0067] The first is a concatenated variant that connects the original features with the fused features. The advantage is high computational efficiency, but the disadvantage is that the feature representation information is not comprehensive. It is suitable for scenarios where feature information is relatively sparse.

[0068] The second is a variant of Transformer, which merges the original hidden state and the fused hidden state, and uses the Transformer encoder to further process these merged hidden states. The advantage is rich feature information, but the disadvantage is poor stability. It is suitable for scenarios with high feature dimensions and large data volumes.

[0069] The combined features are passed through a linear layer and then concatenated with features from other modalities, preserving the integrity of the original signal while integrating rich cross-model information.

[0070] Step 5: Context modeling.

[0071] Contextual data (previous utterances) are integrated into the existing model framework, and two different context integration methods are compared.

[0072] The first method is to connect the context (previous utterance) and the current utterance as a single input stream of the model. The advantages are efficient calculation and simple model building, but the disadvantage is that information is easily lost. It is suitable for scenarios with relatively simple text information.

[0073] The second method is to process the context and the current utterance independently, and then fuse their respective representation outputs. The advantage is that the information can be accurately represented, but the disadvantage is that the modeling is complex. It is suitable for scenarios with complex semantic information.

[0074] Best practices for specific applications

[0075] Case 1: Substation equipment monitoring

[0076] In the substation, audio acquisition equipment and related sensors are installed to collect the operating audio signals of transformers, circuit breakers and other equipment in real time, and record the corresponding text information at the same time, including the operating status of the equipment, whether there is abnormal noise, fault evaluation and subsequent processing suggestions. These data are transmitted to the monitoring system and analyzed using the technology of the present invention. First, a mechanism similar to the self-attention encoder is adopted through the cross-attention encoder, using the query of one model and the key and value of another model to capture the interdependence between different modalities such as audio and text; then, after the cross-attention encoder, the connection across time steps of the new feature representation is modeled to provide information input for the point-by-point feedforward network; then the information of the self-attention encoder is accessed, and the fully connected feedforward network and ReLU activation function are applied to each individual position to refine the encoded feature representation. When a fault is detected in the equipment, the system can issue an alarm in time and provide detailed information of the fault, such as the fault type, the location of occurrence, etc., as well as fault handling suggestions, so that the operation and maintenance personnel can quickly take corresponding treatment measures to ensure the safe and stable operation of the substation.

[0077] Case 2: Power plant equipment monitoring

[0078] In power plants, audio and text information is collected for key equipment such as generator sets and steam turbines. By analyzing the sound characteristics and related text records of the equipment during operation, such as the vibration of the equipment, whether there is abnormal noise, fault history, etc., a mechanism similar to the self-attention encoder is first used through the cross-attention encoder, using the query of one model and the key and value of another model to capture the interdependence between different modalities such as audio and text; then after the cross-attention encoder, the connection across time steps of the new feature representation is modeled to provide information input for the point-by-point feedforward network; then the information of the self-attention encoder is connected, and the fully connected feedforward network and ReLU activation function are applied to each individual position to refine the encoded feature representation. Finally, potential equipment failures are discovered in a timely manner, and preventive maintenance measures are taken in advance to avoid downtime accidents caused by equipment failures and ensure the normal power generation operation of the power plant.

[0079] The present invention also provides an electric power equipment anomaly detection system based on an audio-text cross-modal multi-loss fusion network, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0080] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for detecting abnormality of power equipment based on audio-text cross-modal multi-loss fusion network, characterized in that: A fusion network is designed to achieve the fusion of multimodal data and is trained using a multi-loss training mechanism to achieve anomaly detection of power equipment.

2. According to claim 1, a method for detecting abnormalities in power equipment based on an audio-text cross-modal multi-loss fusion network is characterized in that: Before multimodal fusion of data, feature extraction is performed through the feature network.

3. According to claim 2, a method for detecting abnormalities in power equipment based on an audio-text cross-modal multi-loss fusion network is characterized in that: The feature network includes a pre-trained text feature network and an audio feature network. The text feature network uses RoBERTa, and the audio feature network uses a long short-term memory network.

4. According to claim 1, a method for detecting abnormalities in power equipment based on an audio-text cross-modal multi-loss fusion network is characterized in that: The fusion network includes a cross-attention encoder, a self-attention encoder, and a point-by-point feedforward network, where The criss-cross attention encoder adopts a mechanism similar to the self-attention encoder, using the query from one model and the key and value from another model to capture the interdependencies between different modalities; A self-attention encoder is used to model the connections across time steps of the new feature representation after passing through the criss-cross attention encoder, thereby providing informative input to the point-by-point feedforward network; A point-by-point feedforward network accesses the information from the self-attention encoder, applies a fully connected feedforward network and ReLU activation function to each individual position, and refines the encoded feature representation.

5. According to claim 4, a method for detecting abnormalities in power equipment based on an audio-text cross-modal multi-loss fusion network is characterized in that: The criss-cross attention encoder is described by: in: Q m1 =W q ·f m1 K m2 =W k ·f m2 V m2 =W v ·f m2 The query is represented as Q m1 , which comes from Model 1; denote the key and value as K m2 and V m2 , which comes from model 2; f m1 is the feature of model 1, f m2 is the characteristic of model 2, W q , W k W v Corresponding to different features f m1 、f m2 The weight of is the visual modality feature, d k is the scaling factor.

6. According to claim 3, a method for detecting abnormalities in power equipment based on an audio-text cross-modal multi-loss fusion network is characterized in that: The multi-loss training mechanism is used for training, which is implemented as follows: An additional fully connected layer is added between the text feature network and the fusion network, and between the audio feature network and the fusion network; three different loss functions are applied during training, each corresponding to an output, and the final loss is the sum of the loss from the subnet and the loss from the fusion network: Loss=∑α m *lossfunction(y m ,target m ) Among them, α m is the weight parameter, lossfunction is the loss function equation, y m is the output calculated by the network, target m is the target label.

7. The method for detecting abnormality of power equipment based on an audio-text cross-modal multi-loss fusion network according to claim 1 is characterized in that: Also included are original signal recovery variants designed to mitigate the loss of original signal during cross-model projection.

8. The method for detecting abnormality of power equipment based on an audio-text cross-modal multi-loss fusion network according to claim 7 is characterized in that: The original signal recovery variants include: Concatenation variant, concatenating the original features with the fused features; Transformer variant that merges the original hidden state and the fused hidden state and processes the merged hidden state using the Transformer encoder; The combined features are passed through a linear layer and concatenated with features from other modes, maintaining the integrity of the original signal while integrating cross-model information.

9. A power equipment anomaly detection system based on audio-text cross-modal multi-loss fusion network, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images