A fault identification method based on a dynamic multi-modal fusion network

Abstract

The application relates to a fault identification method based on a dynamic multi-modal fusion network and belongs to the fault identification field.The method comprises the following steps: constructing a multi-modal fusion network based on fault identification of different granularity levels; training and optimizing the multi-modal fusion network based on fault identification of different granularity levels to realize fault identification of different granularity levels.In the reasoning training process, a data-dependent forward path is generated, the problem of resource waste caused by the difficulty of balancing different computing requirements of different modes in the traditional static multi-modal fusion method is solved, the model computing efficiency is improved, and the computing cost is saved; for fault identification problems of different granularity levels, two DynMM algorithms based on a gating network are proposed, so that the algorithms can not only solve simple fault identification tasks such as binary classification, but also be used for completing more difficult multi-modal fusion fault identification problems similar to semantic segmentation, and the universality and identification precision of the model are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of fault identification, and particularly relates to a fault identification method based on a dynamic multi-modal fusion network. BACKGROUND

[0002] In recent years, with the optimization of the power system and the explosive growth of a large number of new energy systems, the power equipment, as a large industrial equipment, has a high price and its health condition gradually degrades over time. Moreover, due to the difficulty and high cost of collecting fault samples, the research in the field of power equipment fault identification is challenged. Meanwhile, the power equipment fault has high complexity and correlation, and even a local fault can cause a global abnormality. Therefore, how to accurately identify and locate the fault by using the multi-modal data generated by the power equipment in the online running state has become a difficult problem to be solved in the field of power equipment fault identification. SUMMARY

[0003] The application aims to solve the problem that single modal data cannot achieve accurate power equipment fault identification, and provides a fault identification method based on a dynamic multi-modal fusion network, so as to improve the model calculation efficiency, save the calculation cost, and improve the generality and identification accuracy of the model.

[0004] To achieve the above object, the technical scheme of the application is as follows: a fault identification method based on a dynamic multi-modal fusion network, comprising:

[0005] a multi-modal fusion network based on fault identification of different granularity levels is constructed;

[0006] the multi-modal fusion network based on fault identification of different granularity levels is trained and optimized to realize fault identification of different granularity levels.

[0007] In an embodiment of the application, the multi-modal fusion network based on fault identification of different granularity levels comprises a modal level decision DynMM unit and a fusion level decision DynMM unit.

[0008] In an embodiment of the application, the modal level decision DynMM unit is specifically implemented as follows:

[0009] Supposing that there are M modalities of input data, the input representation is x=(x1, x2,..., xM), and the output representation is y=(y1, y2,..., yK). M );

[0010] A group of expert networks is designed according to the expert mixed MoE framework, and each expert network studies one of the M modalities.

[0011] A gating network G(x) is proposed to determine the expert network to be activated, the gating network G(x) forms a global view using the multi-modal input x, and then generates a B-dimensional sparse vector g as an output, and the final output y is in the form of:

[0012]

[0013] Wherein, B is the number of selected expert networks, g i is a sparse vector, x i represents the i-th expert as the input of the modal subset, E i is the i-th expert network;In the above formula, the sparse vector g outputs one-hot encoding, that is, only one branch needs to be selected for each fault instance, therefore, only one branch is selected for calculation at each time of network forward transmission.

[0014] In an embodiment of the present application, the input of the gating network G(x) includes the input level features, the intermediate features of each modal.

[0015] In an embodiment of the present application, the fusion level decision DynMM unit is implemented as follows:

[0016] Construct a fusion unit: assuming that the input data has M modalities, that is, x=(x1,x2,...,x M ), a set of fusion operations is represented as {O i}, O i The fusion operation is realized as any function to fuse multi-modal features;The output h of the fusion unit is represented as:

[0017]

[0018] Wherein, B is the total number of operations, O i is the i-th fusion operation;The separate gating network G(x) in each fusion unit accepts multi-modal input and generates a B-dimensional sparse vector g to determine the execution of the corresponding fusion operation, and the sparse vector g outputs one-hot encoding;

[0019] By stacking the fusion unit and interleaving the static feature extraction block, the fusion level decision DynMM unit is constructed, and the separate gating network G(x) in each fusion unit is replaced by a global gating network

[0020] In an embodiment of the present application, the fusion unit has two input modes: x=(x1,x2) and three operations: O1=x1, O2=x1+x2, O3=w1x1+w2x2, wherein w1 and w2 are learnable parameters.

[0021] In one embodiment of the present invention, a resource-aware loss function is introduced into the training of a multimodal fusion network for fault identification at different granularities:

[0022]

[0023] Among them, C(E) i ) represents the execution expert network E i The computational cost, L task Let λ represent the task loss, and let λ be a hyperparameter that controls the relative importance of the two loss terms.

[0024] In one embodiment of the present invention, a resource-aware loss function is introduced into the training of a multimodal fusion network for fault identification at different granularities:

[0025]

[0026] Among them, C(O) i L(j) represents the computational cost of the i-th fusion operation in the j-th unit. task Indicates mission loss. Let F represent the decision vector given by the j-th fusion unit, where F is the number of fusion units and λ is a hyperparameter that controls the relative importance of the two loss terms.

[0027] In one embodiment of the present invention, a multimodal fusion network based on fault identification at different granularities is trained and optimized, specifically as follows:

[0028] Step 1, Pre-training: For modal-level decision-making DynMM units, fully train each expert network of the modal-level decision-making DynMM unit; for fusion-level decision-making DynMM units, use random decision-making for each fusion unit of the fusion-level decision-making DynMM unit.

[0029] The second step, fine-tuning: During training, two training techniques are employed: First, using soft tuning in the resource-aware function. To replace hard g, a method is used to achieve backpropagation; secondly, following the straight-through technique, a hard g is used in forward propagation and a soft g is used in backward propagation. The gradient is approximated as The network parameters were updated using two training techniques.

[0030] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0031] Compared with the prior art, the present application has the following beneficial effects: the present application generates a data-dependent forward path in the reasoning training process, solves the resource waste problem caused by the difficulty of traditional static multi-modal fusion methods in balancing different computing needs of different modalities, improves the model computing efficiency, and saves the computing cost; for fault identification problems of different granularity levels, two DynMM algorithms based on gating network are proposed, which can not only solve simple fault identification tasks such as binary classification, but also be used for more difficult multi-modal fusion fault identification problems similar to semantic segmentation, and improve the generality and identification accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The figure is a schematic diagram of a modal level DynMM unit of the present application containing two modal input data.

[0033] Figure 2 The figure is a design of a fusion level DynMM unit of the present application containing two modal input data.

[0034] Figure 3 The figure is a dynamic multi-modal architecture with a stacked fusion unit. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be specifically described below in combination with the drawings.

[0036] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0037] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0038] The present application provides a fault identification method based on a dynamic multi-modal fusion network, comprising:

[0039] Constructing a multi-modal fusion network based on fault identification of different granularity levels;

[0040] Training and optimizing the multi-modal fusion network based on fault identification of different granularity levels to realize fault identification of different granularity levels.

[0041] The following is the specific implementation process of the present application.

[0042] AsFigure 1 As shown, this embodiment provides a fault identification method based on a dynamic multimodal fusion network, including: First, introducing a decision-making scheme based on a gating network, considering both modal-level (coarse-level) and fusion-level (fine-level) decisions, enabling DynMM to generate data-dependent forward paths during inference training; Second, proposing a training loss function that considers resource budget, thereby reducing decision bias during DynMM training; Finally, implementing non-differentiable gating network optimization, enabling DynMM to backpropagate, thus completing end-to-end training. The specific steps are as follows:

[0043] Step 1: Modal Level Decision (DynMM).

[0044] First, suppose the input data has M modes, then the input is represented as x = (x1, x2, ..., x...). M Following the classic Mixture-of-Experts (MoE) framework, a set of expert networks is designed, with each network specializing in a subset of the M modes. For example, if M=3, there can be a maximum of 7 expert networks, denoted as E1(x1), E2(x2), E3(x3), E4(x1,x2), E5(x2,x3), E6(x1,x2), and E7(x1,x2,x3). In practical applications, domain expertise can be used to narrow down the pool of candidate expert networks.

[0045] This paper proposes a gated network G(x) to determine which expert network should be activated. This gated network uses multimodal input x to form a global view and then generates a B-dimensional sparse vector g as the output. The final output y has the following form:

[0046]

[0047] Where B represents the number of expert networks selected. i It is a sparse vector. i E represents the modality subset of the i-th expert as input. i For expert network.

[0048] Unlike traditional MoE, where the output is a weighted sum of the expert networks and each branch needs to be executed, the g output of the gated network in the above formula is a one-hot encoding, meaning that only one branch needs to be selected for each fault instance. Therefore, only one branch is selected for computation during each forward pass of the network, thus maximizing computational efficiency. Figure 1 For a modal-level DynMM with two modal input data, the dashed line indicates that the expert network is not selected for activation, and the solid line indicates that it is activated.

[0049] The gated network G(x) is not limited to acquiring input-level features; it can also take intermediate features of each modality as network input, allowing modal-level DynMM to be inserted into any part of the multimodal network, saving computational resources after the gated network.

[0050] Step 2: Fusion-level decision DynMM.

[0051] First, construct the fusion unit. Assume the input data has M modes, i.e., x = (x1, x2, ..., x...). M A set of fusion operations is represented as {O ). i}. O i It can be implemented as any function to fuse multimodal features, such as a simple identity mapping (i.e., O(n)). i =x1), addition (i.e., O) i = x1 + x2 + ... + x M ), series (i.e. O) i = [x1, x2, ..., x M ]) and self-attention. Figure 2 An example design of the fusion unit is shown, with two input modes (i.e., x = (x1, x2)) and three operations (i.e., O1 = x1, O2 = x1 + x2, O3 = w1x1 + w2x2), where w1 and w2 are learnable parameters.

[0052] In practice, it can employ more complex fusion operations in each unit to expand its representational capacity. The output of unit h can be expressed as:

[0053]

[0054] Where B represents the total number of operations. i Let g be the i-th fusion operation. The gating network G(x) accepts multimodal input and generates a B-dimensional vector g that determines which operation to perform. According to Step 1, g is used as a one-hot code to improve computational efficiency.

[0055] The fused DynMM unit constructs a dynamic network by stacking fused units, which supports making operational decisions in a more granular and flexible manner. Figure 3 (a) provides an example architecture with two modalities of input. The network consists of four fusion blocks and a global gating network, allowing for flexible control of the fusion level on a sample-by-sample basis. The yellow and green highlighted blocks are static feature extraction blocks, used alternately with the fusion units.

[0056] Figure 3(b) shows the final architecture when the gating network assigns O2 operation to fusion units 1 and 2 and O1 operation to fusion units 3 and 4, which not only skips the complex fusion operation in the unselected fusion unit but also saves unnecessary computation in the feature extraction layer. Since we only use the features of modality 1 after fusion unit 2, there is no need to process the features of modality 2. Therefore, we can skip the computation in the feature extraction layer of x2 (block 3-4 marked in gray). In essence, when combining low-level features from each modality (i.e., early-stage fusion) is sufficient to achieve good prediction, the fusion-level DynMM can save future fusion and modality operations for certain multi-modal inputs. On the other hand, for difficult-to-analyze fault instances, DynMM provides the option to combine multi-modal features in each unit with complex fusion operations to achieve maximum expressiveness. Here, the four separate gating networks G(x) in each fusion unit are replaced by a global gating network to achieve better integration; with multi-modal features (x1, x2) as input and deciding which fusion operation to use for the four fusion units.

[0057] Step 3: Resource-aware loss function.

[0058] For both modal-level and fusion-level DynMM designs, the computation of each expert network E i (or operation O i ) is different. Generally, the expert network (or operation) with more computation has stronger expressiveness. If the network is trained directly by minimizing the task-specific loss, the gating network will always choose the branch with more computation as the solution, which is often a simple solution. To achieve effective reasoning, a resource-aware loss function is introduced in training, as follows:

[0059] (modal level)

[0060] (fusion level)

[0061] where C(E i ) represents the computation cost of executing expert network E i . C(O i , j) represents the computation cost of the i-th fusion operation in the j-th unit. L task represents the task loss, such as the cross-entropy between the network prediction and the true label for classification. g(j) represents the decision vector given by the j-th fusion unit. B is the total number of experts (operations), and F is the number of fusion units. λ is a hyperparameter that controls the relative importance of the two loss terms. The computation cost can be set before training and is a constant term.

[0062] Both the modal-level and fusion-level resource-aware loss functions take into account the computational cost of executing each path, enabling DynMM to achieve the desired trade-off between accuracy and efficiency. Moreover, the value of λ can be adjusted according to the constraints on the amount of computation that can be deployed. For a larger λ, DynMM will prioritize lightweight computations for high computational efficiency. For a smaller λ, DynMM will explore these computationally heavy paths more frequently, resulting in higher precision.

[0063] Step 4: Optimization of non-differentiable gating network.

[0064] DynMM requires training in an end-to-end manner. The current gating network provides discrete decisions, so branch selection cannot be directly differentiated with respect to the gating network. To this end, Gumbel-softmax and a network parameter update method are introduced during training. As shown in Step 1, g represents the required one-hot encoded B-dimensional decision vector generated by the gating network G(x), i.e.:

[0065] g = one-hot(argmax i G(x) i )

[0066] Therefore, a real-valued soft vector is selected to implement network backpropagation, which can be expressed as follows:

[0067]

[0068] where b1, b2,..., b B are independently sampled from Gumbel(0, 1), and τ represents the temperature coefficient of the softmax function. When τ is large, the distribution of is more uniform, and when τ is small, the distribution is similar to the classification distribution.

[0069] During training, two training techniques are used: first, soft is used instead of hard g in the resource-aware function to enable backpropagation. During training, we reinforce τ so that gradually converges to the required one-hot vector. Second, the straight-through technique is followed, using hard g in forward propagation and soft in backward propagation. The gradient approximation is In this way, the gate network still outputs discrete decisions during training. To make the computation more advantageous, the hard g will always be used during inference. Next, we propose a two-stage training for DynMM, which jointly optimizes the multi-modal network and the gating network module to complete the update of network parameters, as follows:

[0070] Step 1: Pre-training. In the early stage of training, the biased optimization caused by the sparse decision of the gating network may occur. And the branches that are selected less have less and smaller weight updates, thus performing poorly and leading to less selection, which will be difficult to be further optimized. The goal of the pre-training stage is to ensure that each branch of DynMM can be fully optimized before the intervention of the gating module. For the modal-level DynMM, we fully train each expert network in this stage. For the fusion-level DynMM, we use random decision (i.e., randomly select an operation from the candidate operation set) for each fusion unit, so that each path of the dynamic network is uniformly optimized.

[0071] Step 2: Fine-tuning. Update the network parameters by the above two training techniques to jointly optimize the dynamic network and the gating network in an end-to-end manner.

[0072] Step 5: Realize fault identification based on the trained DynMM network.

[0073] After the training of the fault identification model based on DynMM is completed, the obtained multi-modal related data is analyzed by the trained network model to obtain the fault identification result.

[0074] The application also provides a computer readable storage medium, which stores computer program instructions capable of being run by a processor, and when the processor runs the computer program instructions, the method steps as described above can be realized.

[0075] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0076] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0077] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0078] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0079] The above description is only preferred embodiments of the present application, not intended to limit other forms of the application. Any person familiar with the art can make changes or modifications to the above-mentioned technical content of the disclosure as equivalent embodiments. However, any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A fault identification method based on a dynamic multi-modal fusion network, characterized in that, The application relates to a multi-modal fusion network based on power equipment fault recognition of different granularity levels. The multi-modal fusion network based on power equipment fault recognition of different granularity levels comprises a modal level decision DynMM unit and a fusion level decision DynMM unit. The multi-modal fusion network based on power equipment fault recognition of different granularity levels is trained and optimized to realize fault recognition of different granularity levels. The modal level decision DynMM unit is specifically implemented as follows:

2. The fault identification method based on the dynamic multi-modal fusion network according to claim 1, characterized in that, An expert network is designed according to an expert mixed MoE framework, and each expert network studies one of M modal subsets; Let the input data have M modalities, the input representation is x = (x1, x2,..., x M ). A gating network G(x) is proposed to determine the activated expert network, the gating network G(x) forms a global view by using the multi-modal input x, and then generates a B-dimensional sparse vector g as an output, and finally the output y is in the form of: The input of the gating network G(x) comprises input level features and intermediate features of each modal. where B is the number of selected expert networks, g i is a sparse vector, x i denotes the subset of modalities that the ith expert takes as input, E i is the ith expert network; in the above formula, the sparse vector g outputs one-hot encoding, that is, each fault instance only needs to select one branch, so only one branch is selected for calculation at each network forward pass.

3. The fault identification method based on the dynamic multi-modal fusion network according to claim 2, characterized in that, The fusion level decision DynMM unit is specifically implemented as follows:

4. The fault identification method based on the dynamic multi-modal fusion network according to claim 1, characterized in that, The fusion level decision DynMM unit is constructed by stacking fusion units and interlacing static feature extraction blocks, and meanwhile, the individual gating network G(x) in each fusion unit is replaced by a global gating network Ĝ(x). Constructing a fusion unit: Suppose the input data has M modes, i.e., x = (x1, x2, ..., x...). M ), representing a set of fusion operations as {O i }, O i The fusion operation is implemented as any function to fuse multimodal features; the output h of the fusion unit is expressed as: Wherein, B is the total number of operations, O i is the i-th fusion operation; the individual gating network G(x) in each fusion unit accepts multi-modal input and generates a B-dimensional sparse vector g that decides to perform the corresponding fusion operation, and the sparse vector g outputs one-hot encoding; The fusion unit has two input modes: x = (x1, x2) and three operations: O1 = x1, O2 = x1 + x2 and O3 = w1x1 + w2x2, wherein w1 and w2 are learnable parameters.

5. The fault identification method based on the dynamic multi-modal fusion network according to claim 4, characterized in that, A resource-aware loss function is introduced in the training of the multi-modal fusion network for fault recognition of different granularity levels.

6. The fault identification method based on the dynamic multi-modal fusion network according to claim 2, characterized in that, A resource-aware loss function is introduced in the training of the multi-modal fusion network for fault recognition of different granularity levels. where C(E i ) denotes the computational cost of executing the expert network E i , L task denotes the task loss, and λ is a hyperparameter that controls the relative importance of the two loss terms.

7. The fault identification method based on the dynamic multi-modal fusion network according to claim 4, characterized in that, The multi-modal fusion network based on power equipment fault recognition of different granularity levels is trained and optimized, and the specific implementation is as follows: where C(O i , j) denotes the computation cost of the i-th fusion operation in the j-th unit, L task denotes the task loss, denotes the decision vector given by the j-th fusion unit, F is the number of fusion units, and λ is a hyper-parameter that controls the relative importance of the two loss terms.

8. The fault identification method based on the dynamic multi-modal fusion network according to claim 1, characterized in that, In the first step, pre-training, each expert network of the modal level decision DynMM unit is fully trained, and each fusion unit of the fusion level decision DynMM unit adopts random decision. 9.A computer readable storage medium having stored thereon computer program instructions capable of being run by a processor, when the processor runs the computer program instructions, the computer program instructions can realize the method steps of any one of claims 1-8. Second step, fine-tuning: during the training process, two training techniques are adopted: one is to use soft g instead of hard g in the resource-aware function to realize back propagation; the second is to follow the straight-through technique, that is, to use hard g in forward propagation and to use soft g in backward propagation, and the gradient approximation is ; the network parameter is updated through the two training techniques. ​

Images